Information processing method and apparatus for calculating information regarding measurement target on the basis of captured images

ABSTRACT

From a sequence of images captured by an image pickup unit, images necessary for measuring placement information regarding markers and/or a sensor are automatically determined and obtained. To this end, using position and orientation information regarding the image pickup unit at the time the image pickup unit has captured an obtained image and placement information regarding detected markers, whether to use the captured image corresponding to the position and orientation is determined. Using the captured image determined to be used, the marker placement information, placement information regarding a measurement target, or the position and orientation of the image pickup unit serving as an unknown parameter is obtained so as to minimize the error between the measured image coordinates and theoretical image coordinates of each marker, which are estimated on the basis of a rough value of the parameter.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to techniques for calculating, usingcaptured images of markers existing in a real space, placementinformation regarding the markers or a measurement target or a parameterof an image pickup unit.

2. Description of the Related Art

Extensive research has been conducted on the mixed reality (MR)technology that superposes virtual space information (e.g., virtualobjects rendered by computer graphics (CG) or text information) on areal space in real time and presents the superposed result to a user. Animage display apparatus used in such an MR system is mainly implementedby a video see-through method of superposing an image in a virtualspace, which is generated in accordance with the position andorientation of an image pickup device, such as a video camera, on animage in a real space, which is captured by the image pickup device, andrendering and displaying the superposed images.

To enable the user to use the MR system without any uncomfortablefeeling, an important factor lies in how accurately the real space isregistered with the virtual space. Many measures have been taken toachieve the accurate registration. In general, the registration problemin MR applications eventually leads to the problem of obtaining theposition and orientation of the image pickup device relative to a spaceor an object onto which virtual information is to be superposed.

To solve the problem, registration techniques using markers placed orset in an environment or on an object are disclosed in“Fukugo-genjitsukan ni okeru ichiawase shuhou (A review of registrationtechniques in mixed reality)” by Sato and Tamura, Collected Papers I ofMeeting on Image Recognition and Understanding (MIRU 2002), InformationProcessing Society of Japan (IPSJ) Symposium Series, vol. 2002, no. 11,pp. I.61-I.68, 2002 (hereinafter referred to as “document 1”). In thesetechniques, the three-dimensional coordinates of each maker are given inadvance. The position and orientation of an image pickup device arecomputed using the relationship between the preliminary-giventhree-dimensional coordinates of each marker and the image coordinatesof each marker within an image captured by the image pickup device.

In “Fukugo-genjitsukan no tame no haiburiddo ichiawase shuhou-6 jiyudosensa to bijon shuhou no heiyou—(A robust registration method formerging real and virtual worlds-combining 6 degree of freedom (DOF)sensor and vision algorithm)” by Uchiyama, Yamamoto, and Tamura,Collected Papers of the Virtual Reality Society of Japan, vol. 8, no. 1,pp. 119-125, 2003 (hereinafter referred to as “document 2”), a hybridapproach utilizing markers and a magnetic or optical 6-DOFposition/orientation sensor is disclosed. Although the 6-DOFposition/orientation sensor provides measured values in a stable manner,the accuracy is not sufficient in many cases. Therefore, in this method,the position and orientation of an image pickup device, which areobtained from the 6-DOF position/orientation sensor mounted on the imagepickup device, are corrected using image information regarding themarkers, thereby improving the accuracy.

In “UG+B hou: shukan oyobi kyakkan shiten kamera to shisei-sensa womochiita ichiawase shuhou (UG+B: A registration framework usingsubjective and objective view camera and orientation sensor) by Sato,Uchiyama, and Yamamoto, Collected Papers of the Virtual Reality Societyof Japan, vol. 10, no. 3, pp. 391-400, 2005 (hereinafter referred to as“document 3”), a 3-DOF orientation sensor is mounted on an image pickupdevice, and a method of measuring the position and orientation of theimage pickup device using image information regarding markers and anorientation value measured by the orientation sensor is disclosed.

In the above-described registration methods using the markers,three-dimensional information regarding the markers (hereinafterreferred to as “placement information regarding the markers” or simplyas “marker placement information”) in a three-dimensional coordinatesystem serving as a registration reference (hereinafter referred to as a“reference coordinate system”) needs to be obtained in advance. Theplacement information regarding the markers can be manually measuredusing a ruler, a protractor, a meter, and the like. However, such manualmeasurement involves many steps but has poor accuracy. Thus, the markerplacement information has been obtained in a simple, highly accuratemanner using image information.

Placement information regarding markers, each of the markers beingrepresented as the position of a point in a three-dimensional space(hereinafter referred to as “point markers”), can be obtained using abundle adjustment method. The bundle adjustment method is a method ofsimultaneously calculating, on the basis of many images captured fromvarious directions, the positions of a group of points in the space andthe position and orientation of the image pickup device at the time theimage pickup device has captured each of the images. More specifically,the method optimizes the positions of the group of points and theposition and orientation of the image pickup device so as to minimizethe sum of errors between the observed positions of the points in eachof the captured images and the calculated positions of the points in theimage, which are calculated on the basis of the positions of the pointsand the position and orientation of the image pickup device. Incontrast, as in the case of a marker with a two-dimensional shape, suchas a square marker, whose placement information is represented by theposition and orientation in the reference coordinate system (hereinafterreferred to as a “two-dimensional marker”), the bundle adjustmentmethod, which is the method of obtaining the positions of the group ofpoints, cannot be applied directly. Therefore, a method of obtainingplacement information regarding two-dimensional markers and pointmarkers, which is similar to the known bundle adjustment method, isdisclosed in “Maka haichi ni kansuru senkenteki chishiki wo riyoushitamaka kyaribureishon houhou (A marker calibration method utilizing apriori knowledge on marker arrangement)” by Kotake, Uchiyama, andYamamoto, Collected Papers of the Virtual Reality Society of Japan, vol.10, no. 3, pp. 401-410, 2005 (hereinafter referred to as document 4).

In the above-described registration methods using the markers and the6-DOF position/orientation sensor or the 3-DOF orientation sensor, notonly the placement information regarding the markers, but also placementinformation regarding the sensor must be measured in advance.

For example, in the case that the magnetic 6-DOF position/orientationsensor is used, a transmitter is fixed in the space, and a receiver ismounted on a measurement target (e.g., an image pickup device) tomeasure the position and orientation of the measurement target. Thesensor is configured to measure the 6-DOF position/orientation of thereceiver in a coordinate system defined by the transmitter. Thus, toobtain the position and orientation of the measurement target in thereference coordinate system, placement information (that is, theposition and orientation) of the transmitter relative to the referencecoordinate system and placement information (that is, the position andorientation) of the receiver relative to the measurement target must bemeasured in advance. Japanese Patent Laid-Open No. 2003-269913(corresponding to U.S. Pat. No. 6,792,370) discloses a method ofobtaining placement information of the sensor using a plurality ofimages of markers placed in the reference coordinate system, which arecaptured from various directions. In the case that the 3-DOF orientationsensor is used, the orientation sensor is mounted on a measurementtarget (e.g., an image pickup device) to measure the orientation of themeasurement target. To this end, placement information (that is, theorientation) of the orientation sensor relative to the measurementtarget needs to be measured in advance. Japanese Patent Laid-Open No.2005-326275 (corresponding to U.S. Published Application No.2005/0253871) discloses a method of obtaining the orientation of thesensor relative to the measurement target using a plurality of capturedimages of markers.

In the related art, many images captured from various directions havebeen used to measure the placement information regarding the markers forregistration and the sensor. Normally, these images are manuallycaptured by the user who decides the image capturing positions. However,such random capturing of the images is insufficient for highly accuratemeasurement of the placement information. The user must be fullyexperienced and have enough knowledge. In other words, not everyone caneasily measure the placement information.

Since the known measurement of the placement information regarding themarkers and the sensor requires time-consuming preparations, themeasurement must be conducted before the user experiences the MR system.Therefore, no new markers can be added to expand the moving range whilethe user is experiencing the MR system. When the marker or sensorplacement information changes while the user is experiencing the MRsystem, no actions can be taken in real time to handle such a change.

SUMMARY OF THE INVENTION

The present invention allows automatic determination and obtaining ofimages necessary for measuring placement information regarding markersor a sensor from an image sequence captured by an image pickup device,thereby avoiding dependence on user experience or knowledge.

The present invention also allows measurement of placement informationregarding markers or a sensor while a user is experiencing an MR systemby automatically determining and obtaining images necessary formeasuring the placement information regarding the markers or the sensorfrom an image sequence captured by an image pickup device.

Aspects of the present invention have the following structure.

An information processing method according to an aspect of the presentinvention is an information processing method of calculating informationregarding a measurement target using captured images of markers existingin a real space, including the steps of obtaining an image captured byan image pickup unit; extracting markers from the captured image;obtaining position and orientation information regarding the imagepickup unit; determining, on the basis of placement informationregarding the markers, which is managed by a marker informationmanagement unit, and the position and orientation information regardingthe image pickup unit, whether to use the captured image correspondingto the position and orientation information to calculate the informationregarding the measurement target; and calculating the informationregarding the measurement target using the captured image in the casethat it is determined to use the captured image.

An information processing method according to another aspect of thepresent invention is an information processing method of calculatinginformation regarding a measurement target using captured images ofmarkers existing in a real space, including the steps of obtaining animage captured by an image pickup unit; extracting markers from thecaptured image; obtaining position and orientation information regardingthe image pickup unit; calculating, on the basis of the position andorientation information regarding the image pickup unit, areainformation regarding an image capturing area in which an image shouldbe captured; and presenting the calculated area information.

An information processing apparatus according to yet another aspect ofthe present invention is an information processing apparatus forcalculating information regarding a measurement target using capturedimages of markers existing in a real space, including the followingelements: a captured image obtaining unit configured to obtain an imagecaptured by an image pickup unit; an extraction unit configured toextract markers from the captured image; a position/orientationinformation obtaining unit configured to obtain position and orientationinformation regarding the image pickup unit; a determination unitconfigured to determine, on the basis of placement information regardingthe markers, which is managed by a marker information management unit,and the position and orientation information regarding the image pickupunit, whether to use the captured image corresponding to the positionand orientation information to calculate the information regarding themeasurement target; and a calculator configured to calculate theinformation regarding the measurement target using the captured image inthe case that the determination unit determines to use the capturedimage.

An information processing apparatus according to a further aspect of thepresent invention is an information processing apparatus for calculatinginformation regarding a measurement target using captured images ofmarkers existing in a real space, including the following elements: acaptured image obtaining unit configured to obtain an image captured byan image pickup unit; an extraction unit configured to extract markersfrom the captured image; a position/orientation information obtainingunit configured to obtain position and orientation information regardingthe image pickup unit; a calculator configured to calculate, on thebasis of the position and orientation information regarding the imagepickup unit, area information regarding an image capturing area in whichan image should be captured; and a presentation unit configured topresent the area information calculated by the calculator.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an exemplary structure of aninformation processing apparatus according to a first exemplaryembodiment of the present invention.

FIG. 2 is a flowchart of a process performed by the informationprocessing apparatus according to the first embodiment.

FIG. 3A illustrates an environment where a plurality of markers areplaced.

FIG. 3B illustrates a quadrangular marker.

FIG. 3C illustrates circular markers.

FIG. 4 is a flowchart of a process performed by an image determinationunit according to the first embodiment.

FIG. 5 illustrates calculation of evaluation values according to thefirst embodiment.

FIG. 6 is a flowchart of a process of obtaining placement informationregarding the markers or the position and orientation of an image pickupunit.

FIG. 7 is a schematic diagram of an exemplary structure of theinformation processing apparatus according to a second exemplaryembodiment of the present invention.

FIG. 8 is a flowchart of a process performed by the informationprocessing apparatus according to the second embodiment.

FIG. 9 is a flowchart of a process performed by the image determinationunit according to the second embodiment.

FIG. 10 is a flowchart of a process performed by the informationprocessing apparatus according to a third exemplary embodiment of thepresent invention.

FIG. 11 illustrates transformation of a marker placed in a referencecoordinate system to a camera coordinate system according to the thirdembodiment.

FIG. 12 is a schematic diagram of an exemplary structure of theinformation processing apparatus according to a fourth exemplaryembodiment of the present invention.

FIG. 13 is a flowchart of a process performed by the informationprocessing apparatus according to the fourth embodiment.

FIG. 14 illustrates an image capturing area according to the fourthembodiment.

FIGS. 15A and 15B illustrate presentation of the image capturing area orthe image pickup position and orientation according to the fourthembodiment.

FIG. 16 is a schematic diagram of an exemplary structure of theinformation processing apparatus according to a fifth exemplaryembodiment.

FIG. 17 is a flowchart of a process performed by the informationprocessing apparatus according to the fifth embodiment.

FIG. 18 is a schematic diagram of an exemplary structure of theinformation processing apparatus according to a seventh exemplaryembodiment.

FIG. 19 is a flowchart of a process performed by the informationprocessing apparatus according to the seventh embodiment.

FIG. 20 is a flowchart of a process performed by the image determinationunit according to the seventh embodiment.

DESCRIPTION OF THE EMBODIMENTS

Preferred exemplary embodiments of the present invention will now hereinbe described in detail below with reference to the accompanyingdrawings.

First Exemplary Embodiment

An information processing apparatus according to a first exemplaryembodiment calculates, using captured images of markers existing in areal space, placement information regarding the markers.

FIG. 1 schematically shows an exemplary structure of an informationprocessing apparatus 1000 configured to calculate placement informationregarding markers according to the first embodiment.

The information processing apparatus 1000 is connected to an imagepickup unit 100, such as a video camera or the like.

An image obtaining unit 1010 obtains images from the image pickup unit100. The image pickup unit 100 captures an image of a space or an objecton which markers are placed and sequentially outputs a sequence ofmoving images via the image obtaining unit 1010 to the informationprocessing apparatus 1000. The image obtaining unit 1010 is, forexample, a video capture card mounted on a personal computer (PC).Instead of obtaining in real time live images captured by the imagepickup unit 100, such as the video camera, the image obtaining unit 1010may obtain a sequence of images captured in the past, which are storedon a storage device (not shown).

A marker extraction unit 1020 extracts the markers from the obtainedimage.

The markers used in the first embodiment will now be described below.

In an environment where the MR can be experienced (hereinafter simplyreferred to as an “environment”) or on an object in the environment, aplurality of four-sided markers (hereinafter referred to as“quadrangular markers”) shown in FIG. 3A are placed. Each of the placedmarkers is represented as P^(k) (k=1, . . . , K_(o)) where K_(o) is thenumber of the placed markers (K_(o)=3 in the example shown in FIG. 3A).As shown in FIG. 3B, each quadrangular marker has an internal patternrepresenting its identifier. From the identifier, each quadrangularmarker can be uniquely identified. Each quadrangular marker P^(k) hasvertices p^(ki) (i=1, . . . , N_(k)) where N_(k) is the number ofvertices constituting the marker P^(k) (since the markers in the firstembodiment are quadrangular, N_(k)=4).

The marker extraction unit 1020 performs the labeling after binarizing acaptured image and extracts, from a region having an area larger than orequal to a predetermined area, an area defined by four straight lines asa marker candidate area. It is then determined whether the candidatearea is a marker area by determining whether the candidate area containsa specific pattern. In the case that the candidate area is determined asa marker area, the internal pattern of the marker area is read to obtainthe direction of the marker within the image and the identifier of themarker, thereby extracting the marker from the obtained image.

The marker P^(k) placed in the environment or on the object is notlimited to a quadrangular marker. The marker P^(k) can have any shape aslong as it is detectable and identifiable in the captured image.

For example, as shown in FIG. 3C, circular point markers havingdifferent colors may be used. In this case, an area corresponding toeach of the marker colors is detected in an image, and the barycentricposition thereof is determined as the detected coordinates of themarkers. Alternatively, feature points (natural feature points) thatoriginally exist in the space and have different texture features mayserve as markers. In this case, the markers are extracted from an imageby applying a template matching to the image to detect any matches tomarker template images stored in advance as known information.

The markers are not limited to those described above and can be of anytype or shape as long as they are fixed in the space and detectable inthe captured image.

A marker management unit 1030 manages, regarding the markers extractedby the marker extraction unit 1020, the identifier k_(n) of eachquadrangular marker P^(kn), the image coordinates u_(Pkni) of eachvertex p^(kni) of each marker P_(kn), and placement informationregarding each marker P^(kn) as marker information. The placementinformation regarding each marker indicates the position and orientationof the marker relative to a reference coordinate system in the case thatthe marker is a quadrangular marker and indicates the position of themarker relative to the reference coordinate system in the case that themarker is a point marker.

The reference coordinate system is defined on the basis of a point inthe environment or on the object, the point serving as the origin, andthree axes orthogonal to one another are defined as an X-axis, a Y-axis,and a Z-axis, respectively. In the first embodiment, a marker whoseplacement information in the reference coordinate system is known isreferred to as a “reference marker”. Images are captured such that areference marker is included in at least one of the images.

An image-pickup-unit-position/orientation obtaining unit 1040 obtainsthe rough position and orientation of the image pickup unit 100 at thetime the image pickup unit 100 has captured each image.

An image determination unit 1050 determines whether to use the imageobtained by the image obtaining unit 1010 to calculate placementinformation regarding each marker.

Using the image determined by the image determination unit 1050 to beused, a marker-placement-information calculator 1060 calculatesplacement information regarding each marker.

Next, a process of calculating the marker placement informationaccording to the first embodiment will be described below with referenceto the flowchart shown in FIG. 2.

In step S2010, the image obtaining unit 1010 obtains an image currentlybeing captured by the image pickup unit 100.

In step S2020, the marker extraction unit 1020 extracts markers from theimage obtained by the image obtaining unit 1010 and registers theidentifier k_(n) of each extracted quadrangular marker P^(kn) and theimage coordinates u^(Pkni) of each vertex p^(kni) in the markermanagement unit 1030 where n (n=1, . . . , N) is an index correspondingto each of the quadrangular markers detected in the image, and Nindicates the total of the detected quadrangular markers.

Also, N_(Total) is the total of the vertices of N quadrangular markersextracted from the image. In the example shown in FIG. 3A, an image of aquadrangular marker with one identifier, a quadrangular marker with twoidentifiers, and a quadrangular markers with three identifiers iscaptured, where N=3. The identifiers k₁=1, K₂=2, and K₃=3 andcorresponding image coordinates u^(Pk1i), u^(Pk2i), and u ^(Pk3i) (i=1,2, 3, and 4) are registered in the marker management unit 1030. In thiscase, N_(Total) is 12 (=3×4).

In step S2030, the image-pickup-unit-position/orientation obtaining unit1040 obtains the rough position and orientation information of the imagepickup unit 100 at the time the image pickup unit 100 has captured theimage currently being calculated. In the first embodiment, the roughposition and orientation of the image pickup unit 100 are obtained,using a known method, from image information regarding markers whoseplacement information in the reference coordinate system is known. Forexample, in the case that markers detected in the image are not coplanarin the space, a direct linear transformation (DLT) method is used toobtain the position and orientation of the image pickup unit 100. In thecase that the detected markers are coplanar, a method using the planehomography is used to obtain the position and orientation of the imagepickup unit 100. These methods of obtaining the position and orientationof the image pickup unit 100 on the basis of the relationship betweenthe image coordinates and the three-dimensional coordinates of aplurality of points are well known in fields of photogrammetry andcomputer vision, and hence detailed descriptions thereof are omitted toavoid redundancy.

The method of obtaining the position and orientation of the image pickupunit 100 is not limited to the above-described methods. Alternatively,for example, a magnetic, ultrasonic, or optical 6-DOFposition/orientation sensor is mounted on the image pickup unit 100, andan output value of the sensor may be used as the rough position andorientation of the image pickup unit 100. Alternatively, as described indocument 3, the rough position and orientation of the image pickup unit100 may be obtained using a 3-DOF orientation sensor and markers.Alternatively, any other known techniques may be used to obtain therough position and orientation of the image pickup unit 100.

In step S2040, evaluation values are computed. In step S2050, it isdetermined on the basis of the evaluation values whether to use theimage obtained by the image obtaining unit 1010 to calculate placementinformation regarding each marker. FIG. 4 is a flowchart of theprocedure performed in steps S2040 and S2050 to determine whether theimage is necessary for calculating placement information regarding eachmarker.

In step S3010, it is determined whether a reference marker is detectedin the obtained image, or whether a marker with rough placementinformation in the reference coordinate system is detected in the image.The reference marker may be defined as a polygonal marker, such as arectangular marker, whose size detected in the image is known, or as atleast three point markers that are not collinear and that have a knownrelative positional relationship. Rough placement information regardinga quadrangular marker other than the reference marker can be obtained bycalculating the plane homography on the basis of the rough position andorientation of the image pickup unit 100 at the time the image pickupunit 100 has captured an image of the marker and the position of eachvertex in the image. In the case that the marker is a point marker, therough position of the marker can be obtained by a stereographic methodusing two captured images in which the rough positions and orientationsof the image pickup unit 100 at the time the image pickup unit 100 hascaptured the images are known.

In the case that no reference marker or no marker with rough placementinformation is detected in the image, it is determined in step S3050that this image will not be used.

In step S3020, it is determined whether a marker to be measured (servingas a measurement target) other than the reference marker is detected. Inthe case that no marker serving as a measurement target is detected, itis determined in step S3050 that this image will not be used.

In step S3030, the marker information managed by the marker managementunit 1030 and the position and orientation of the image pickup unit 100,which are obtained by the image-pickup-unit-position/orientationobtaining unit 1040, are input to the image determination unit 1050. Instep S3040, evaluation values for determining whether to use the imageto calculate the marker placement information are calculated. A methodof calculating the evaluation values will now be described.

FIG. 5 illustrates a method of obtaining the evaluation values fordetermining whether to use an obtained image for measurement.

Prior to this calculation, images determined to be used for measurementare numbered frame numbers 1, 2, m−1, and it is then determined whetherto use the obtained image as an m-th frame. The position of the imagepickup unit 100 at the time the image pickup unit 100 has captured thecurrently obtained image is C_(m)=[C_(mx) C_(my) C_(mz)]^(t). Thebarycentric position in the reference coordinate system is G_(m)=[G_(mx)G_(my) G_(mz)]^(t), which is obtained from placement informationregarding a group of markers detected in the image obtained in the m-thframe.

Given a vector {right arrow over (A)} extending from the positionC_(m-1) of the image pickup unit 100 at the time the image pickup unit100 has captured the (m−1)-th frame to the barycentric position G_(m-1)of the marker group detected in the (m−1)-th frame, and a vector {rightarrow over (B)} extending from the position C_(m) of the image pickupunit 100 at the time the image pickup unit 100 has captured thecurrently obtained image to the barycentric position G_(m-1). The angleθ_(AB) defined by {right arrow over (A)} and {right arrow over (B)} iscalculated:

$\begin{matrix}{\theta_{AB} = {\arccos\frac{\overset{\rightarrow}{A} - \overset{\rightarrow}{B}}{{\overset{\rightarrow}{A}} - {\overset{\rightarrow}{B}}}}} & \left( {1\text{-}1} \right)\end{matrix}$The obtained angle θ_(AB) serves as a first evaluation value. Next, themagnitude ratio between {right arrow over (A)} and {right arrow over(B)} is calculated:

$\begin{matrix}{{\alpha = {\frac{\overset{\rightarrow}{B}}{\overset{\rightarrow}{A}}\left( {{{where}\mspace{14mu}{\overset{\rightarrow}{A}}} > {\overset{\rightarrow}{B}}} \right)}}\alpha = {\frac{\overset{\rightarrow}{A}}{\overset{\rightarrow}{B}}\left( {{{where}\mspace{14mu}{\overset{\rightarrow}{A}}} < \left. \overset{\rightarrow}{B} \right)} \right.}} & \left( {1\text{-}2} \right)\end{matrix}$The obtained α serves as a second evaluation value.

In step S3050, whether to use the obtained image is determined on thebasis of the obtained evaluation values.

In the case that the first evaluation value θ_(AB) satisfies thefollowing:θ_(AB)>θ_(TAB)  (1-3)it is determined that there is a sufficiently great parallax between the(m−1)-th frame and the currently obtained image, and it is thusdetermined to use the obtained image as the m-th frame for calculation.In expression (1-3), θ_(TAB) is a preset threshold value. The fact thatthe evaluation value θ_(AB) exceeds the threshold θ_(TAB) means that theimages of the detected marker group have been captured at differentviewpoints in the m-th frame and in the (m−1)-th frame. The thresholdθ_(TAB) may be changed to any value in accordance with the distributionof the markers in the three-dimensional space or in the environmentwhere the markers are placed. For example, in the case that the markersare placed in a room in which the image capturing range is limited,θ_(TAB) is set to a relatively small value. In the case that θ_(TAB) isset to a small value, however, the parallax becomes small, and hence,the accuracy of the measurement result is reduced. To prevent theaccuracy of the measurement result from degrading, the number of imagesused may be increased, or a high-resolution camera may be used.

In the case that the condition in expression (1-3) is not satisfied, adetermination is performed on the basis of the second evaluation valueα.

In the case that the second evaluation value α satisfies the following:α<α_(T1) or α_(T2)<α (where α_(T1)<α_(T2))  (1-4)it is determined to use the obtained image as the m-th frame forcalculation. Otherwise, the obtained image will not be used. Inexpressions (1-4), α_(T1) and α_(T2) are preset thresholds.

The fact that the second evaluation value satisfies the above conditionmeans that the image capturing distance to the detected marker groupdiffers between the m-th frame and the (m−1)-th frame.

In step S3060, the determination result obtained in step S3050 whetherto use the image or not is output.

Referring back to FIG. 2, in step 2050, if it is determined in stepS3050 to use the currently obtained image and if the number m of imagesdetermined to be used to calculate the marker placement informationsatisfies the following:m≧2  (1-5)then, the flow proceeds to step S2060. Otherwise (if it is determined instep S2040 that the image will not be used or if the number m of imagesdoes not satisfy expression (1-5)), the flow returns to step S2010, anda new image is obtained.

In step S2060, information regarding the images to be used forcalculation is used to calculate the marker placement information.

A process of calculating the marker placement information in step S2060will now be described. FIG. 6 shows the process of calculating themarker placement information.

In step S5010, the rough position and orientation of the image pickupunit 100 at the time the image pickup unit 100 has captured each of theimages, which have been obtained by the above-described method, and therough placement information regarding each rectangular marker serving asa measurement target are input.

In step S5020, the error (residual error) between the detected positionof the marker in the image and the calculated position of the marker inthe image, which is calculated on the basis of the rough position andorientation of the image pickup unit 100 and the rough placementinformation regarding the marker, is calculated.

Given u=[u_(x) u_(y)]^(t) representing the projected position of themarker on the image, a vector s representing the position andorientation of the image pickup unit 100, and a vector a representingthe marker placement information, then the following holds true:u=F(s,a)  (1-6)where F is a function including a viewing transform, which is atransform from the reference coordinate system to a camera coordinatesystem, and a perspective projection transform. Given t=[t_(x) t_(y)t_(z)]^(t) representing the position of the image pickup unit 100 in thereference coordinate system, and ω=[ω_(x) ω_(y) ω_(z)]^(t) representingthe orientation of the image pickup unit 100, then the vector srepresenting the position and orientation of the image pickup unit 100is s=[t_(x) t_(y) t_(z) ω_(x) ω_(y) ω_(z)]^(t). In this case, the vectora is a three-dimensional vector in the case that the marker only hasposition information and is a six-dimensional vector in the case thatthe marker has position and orientation information. The orientation inthe three-dimensional space is represented by a 3×3 rotation transformmatrix. Since the degree of rotational freedom is only three, theorientation can be represented by the three-dimensional vector ω.

In this case, ω employs a 3-DOF orientation representation method andrepresents the orientation using a rotation axis vector and a rotationangle. Given a rotation angle r_(a), r_(a) can be expressed in terms ofω:r _(a)=√{square root over (ω_(x) ²+ω_(y) ²+ω_(z) ²)}  (1-7)

Given a rotation axis vector r_(axis)=[r_(x) r_(y) r_(z)]t, therelationship between r_(axis) and ω can be expressed:[ω_(x)ω_(y)ω_(z)]=[r_(a)r_(x)r_(a)r_(y)r_(a)r_(z)]  (1-8)The relationship between ω (rotation angle r_(a) and rotation axisvector r_(axis)) and the 3×3 rotation transform matrix R can beexpressed:

$\begin{matrix}\begin{matrix}{R = \begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix}} \\{= \begin{bmatrix}{{r_{x}^{2}\left( {1 - {\cos\; r_{a}}} \right)} + {\cos\; r_{a}}} & {{r_{x}{r_{y}\left( {1 - {\cos\; r_{a}}} \right)}} - {r_{z}\sin\; r_{a}}} & {{r_{z}{r_{x}\left( {1 - {\cos\; r_{a}}} \right)}} + {r_{y}\sin\; r_{a}}} \\{{r_{x}{r_{y}\left( {1 - {\cos\; r_{a}}} \right)}} + {r_{z}\sin\; r_{a}}} & {{r_{y}^{2}\left( {1 - {\cos\; r_{a}}} \right)} + {\cos\; r_{a}}} & {{r_{y}{r_{z}\left( {1 - {\cos\; r_{a}}} \right)}} - {r_{x}\sin\; r_{a}}} \\{{r_{z}{r_{x}\left( {1 - {\cos\; r_{a}}} \right)}} - {r_{y}\sin\; r_{a}}} & {{r_{y}{r_{z}\left( {1 - {\cos\; r_{a}}} \right)}} + {r_{x}\sin\; r_{a}}} & {{r_{z}^{2}\left( {1 - {\cos\; r_{a}}} \right)} + {\cos\; r_{a}}}\end{bmatrix}}\end{matrix} & \left( {1\text{-}9} \right)\end{matrix}$Let û be the detected coordinates of the marker in the image. Then, theerror between u and û is calculated:Δu=û−u  (1-10)

In step S5030, a corrected form of the vector s representing theposition and orientation of the image pickup unit 100 and a correctedform of the vector a representing the marker placement information arecalculated such that the error Δu calculated in step S5020 is minimized.The error Δu can be expressed by first-order approximation using avector Δs for correcting the vector s and a vector Δa for correcting thevector a:

$\begin{matrix}{{{\Delta\; u} \approx {\begin{bmatrix}\frac{\partial u}{\partial s} & \frac{\partial u}{\partial a}\end{bmatrix}\begin{bmatrix}{\Delta\; s} \\{\Delta\; a}\end{bmatrix}}}{{where}\begin{bmatrix}\frac{\partial u}{\partial s} & \frac{\partial u}{\partial a}\end{bmatrix}}} & \left( {1\text{-}11} \right)\end{matrix}$is referred to as a Jacobian matrix (or an image Jacobian). Since aspecific method of calculating a Jacobian matrix is known and describedin document 4, a detailed description thereof is omitted to avoidredundancy.

Expression (1-11) is a relational expression regarding a point (a vertexin the case of a quadrangular marker) in an image. In the case thatthere are m images determined to be used for calculation, and that themarker has q vertices, then the following is derived:

$\begin{matrix}{{\Delta\; u} \approx {\begin{bmatrix}\frac{\partial u}{\partial s_{1}} & \cdots & \frac{\partial u}{\partial s_{m}} & \frac{\partial u}{\partial a_{1}} & \cdots & \frac{\partial u}{\partial a_{q}}\end{bmatrix}\begin{bmatrix}{\Delta\; s_{1}} \\\vdots \\{\Delta\; s_{m}} \\{\Delta\; a_{1}} \\\vdots \\{\Delta\; a_{q}}\end{bmatrix}}} & \left( {1\text{-}12} \right)\end{matrix}$which can be simplified as:

$\begin{matrix}{{{\Delta\; u} \approx {J_{w}\Delta\; t}},\left( {{{where}\mspace{14mu} J_{w}} = \begin{bmatrix}\frac{\partial u}{\partial s_{1}} & \cdots & \frac{\partial u}{\partial s_{m}} & \frac{\partial u}{\partial a_{1}} & \cdots & \frac{\partial u}{\partial a_{q}}\end{bmatrix}} \right)} & \left( {1\text{-}13} \right)\end{matrix}$In the case that the number of vertices of all the markers detected inthe images selected to be used for calculation is n, then the followingis obtained:

$\begin{matrix}{\begin{bmatrix}{\Delta\; u_{1}} \\{\Delta\; u_{2}} \\\vdots \\{\Delta\; u_{n}}\end{bmatrix} \approx {\begin{bmatrix}J_{w\; 1} \\J_{w\; 2} \\\vdots \\J_{wn}\end{bmatrix}\Delta\; t}} & \left( {1\text{-}14} \right)\end{matrix}$The unknown parameter Δt to be calculated is obtained by a least squaresmethod in the following manner:

$\begin{matrix}{{\Delta\; t} = {\begin{bmatrix}J_{w\; 1} \\J_{w\; 2} \\\vdots \\J_{wn}\end{bmatrix}^{+}\begin{bmatrix}{\Delta\; u_{1}} \\{\Delta\; u_{2}} \\\vdots \\{\Delta\; u_{n}}\end{bmatrix}}} & \left( {1\text{-}15} \right)\end{matrix}$where + represents a pseudo-inverse matrix.

The calculation of Δt using equation (1-15) is equivalent to solving aredundant system of linear equations relative to an unknown value of thecorrected vector Δt. Therefore, instead of calculating thepseudo-inverse matrix, a different method of solving the system oflinear equations, such as a sweeping-out method, a Gauss-Jordaniteration method, or a conjugate gradient method, may be used to solvethe system of linear equations. In the case that the number of obtainedimages or the number of detected markers is large, Δt can be calculatedat a high speed by a preconditioned conjugate gradient method ofperforming incomplete Cholesky decomposition as preprocessing.

In step S5040, using the obtained Δt, the following are corrected:s _(i)(1≦i≦m) and a _(j)(1≦j≦q)

In step S5050, whether the calculation has converged or not isdetermined using a determination reference, such as whether ΣΔu is lessthan a preset threshold or whether the corrected Δt is less than apreset threshold. If the calculation has not converged, the correctedstate vectors s and a serve as initial values, and the flow returns tostep S5020. The processing from step S5020 to step S5040 is repeated.

If it is determined that the calculation has converged, in step S5060,the marker placement information or the position and orientation of theimage pickup unit 100 are output.

As has been described above, whether to use a captured image formeasurement is automatically determined using the rough placementinformation regarding each marker detected in the image and theevaluation values calculated on the basis of the rough position andorientation of the image pickup unit 100 at the time the image pickupunit 100 has captured the image. Accordingly, many images captured fromvarious directions, which are needed for highly accurate measurement ofthe marker placement information, can be obtained without requiringexperiences or knowledge.

Modification 1-1

In the first embodiment, the first evaluation value is obtained usingthe vector {right arrow over (A)} extending from the position C_(m-1) ofthe image pickup unit 100 at the time the image pickup unit 100 hascaptured the (m−1)-th frame to the barycentric position G_(m-1) of themarker group detected in the (m−1)-th frame, and the vector {right arrowover (B)} extending from the position C_(m) of the image pickup unit 100at the time the image pickup unit 100 has captured the obtained image tothe barycentric position G_(m-1). Alternatively, instead of using asimple barycenter as G_(m-1), the barycentric position weighted inaccordance with the marker detection result may serve as G_(m-1).

The barycentric position is obtained to take into consideration thedistance to each marker serving as a measurement target. In image-basedmeasurement such as stereographic measurement, the measurement accuracyis affected by the distance to a measurement target and the distancebetween image capturing positions. If, for example, among the detectedmarkers, one marker is exceptionally far away or nearby, therepresentative position of measurement targets is significantly affectedby the exceptional marker. To reduce the effect, the exceptional markeris excluded, or the barycenter calculation is calculated by weightingthis exceptional marker less.

The distance 1i (i=1, 2, . . . , N) between the rough position of eachof the markers detected in the (m−1)-th frame and the rough imagecapturing position of the image pickup unit 100 at the time the imagepickup unit 100 has captured the (m−1)-th frame is obtained, where N isthe total of the markers detected in the (m−1)-th frame. In thebarycenter calculation, the weighted barycentric position G_(m-1) isobtained by applying less weight to the exceptional marker. The weightcan be calculated using, for example, a Tukey function, which is oftenused in M-estimation, which is a robust estimation method. In the Tukeyfunction, using a threshold c obtained on the basis of the distance1_(i) (i=1, 2, . . . , N) and the standard deviation of the distance1_(i), a weight W_(i) is computed:

$\begin{matrix}{{{W_{i} = \left( {1 - \left( \frac{l_{i}}{c} \right)^{2}} \right)^{2}},{{{if}\mspace{14mu}{l_{i}}} \leq c}}{{W_{i} = 0},{{{if}\mspace{14mu}{l_{i}}} > c}}} & \left( {1\text{-}16} \right)\end{matrix}$

Given the position t_(i) (i=1, 2, . . . , N) of each marker detected inthe (m−1)-th frame, the weighted barycentric position G_(m-1) isobtained:

$\begin{matrix}{G_{m - 1} = {\frac{1}{N}{\sum\limits_{1}^{N}\;{t_{i}W_{i}}}}} & \left( {1\text{-}17} \right)\end{matrix}$Modification 1-2

The marker placement information or the position and orientation of theimage pickup unit 100 have been obtained in the first embodiment. Incontrast, an internal parameter of the image pickup unit 100 may becalculated as a measurement target.

Given a focal length f, coordinates (u₀, v₀) at the center of the image,scale factors k_(u) and k_(v) in the u and v directions, and a shearcoefficient k_(s), an internal parameter of the image pickup unit 100,which serves as an unknown parameter, is:

$\begin{matrix}{A = \begin{bmatrix}{fk}_{u} & {fk}_{z} & u_{0} \\0 & {fk}_{v} & v_{0} \\0 & 0 & 1\end{bmatrix}} & \left( {1\text{-}18} \right)\end{matrix}$where A is a matrix for transforming a point in the camera coordinatesystem to the image coordinates u.

Whether to use an obtained image for measurement can be determined by amethod similar to that used in the first embodiment. An internalparameter of the image pickup unit 100 serves as an unknown parameter,and a parameter that minimizes the error between the measured imagecoordinates of each marker and the theoretical image coordinates of themarker calculated from an estimated parameter value is obtained bynon-linear optimization. In this case, the rough internal parameter ofthe image pickup unit 100 is set in advance on the basis of a designparameter or the like.

Modification 1-3

In the first embodiment, the first evaluation value θ_(AB) is obtainedfrom the vector {right arrow over (A)} extending from the positionC_(m-1) of the image pickup unit 100 at the time the image pickup unit100 has captured the (m−1)-th frame to the barycentric position G_(m-1)of the marker group detected in the (m−1)-th frame and the vector {rightarrow over (B)} extending from the position C_(m) of the image pickupunit 100 at the time the image pickup unit 100 has captured the obtainedimage to the barycentric position G_(m-1).

However, the first evaluation value need not be obtained only on thebasis of the (m−1)-th frame and the obtained image. Let {right arrowover (A)} be a vector extending from the barycentric position G_(m) ofthe marker group detected in the obtained image to the position C_(m) ofthe image pickup unit 100 at the time the image pickup unit 100 hascaptured the obtained image. Also, images each including markers withthe same identifiers as those of the markers detected in the obtainedimage are extracted from the first to (m−1)-th frames. Given the total Qof the extracted images and a vector {right arrow over (B)}_(q)extending from the barycentric position Gm of the marker group detectedin the obtained image to a position C_(q) of the image pickup unit 100at the time the image pickup unit 100 has captured a q-th frame (q=1, 2,. . . , Q). The first evaluation value is:

$\begin{matrix}{\theta_{{AB}_{q}} = {\arccos\frac{\overset{\rightarrow}{A} \cdot {\overset{\rightarrow}{B}}_{q}}{{\overset{\rightarrow}{A}} \cdot {{\overset{\rightarrow}{B}}_{q}}}}} & \left( {1\text{-}19} \right)\end{matrix}$

In the case that, for all q, the first evaluation value θ_(ABq)satisfies the following:θ_(ABq)>θ_(TAB)  (1-20)then, it is determined that the obtained image will be used forcalculation. Otherwise, the obtained image will not be used forcalculation. In expression (1-20), θ_(TAB) is a preset threshold. Thefact that the evaluation value θ_(ABq) exceeds the threshold θ_(TAB)means that images of the detected marker group have been captured atdifferent viewpoints in the m-th frame and the q-th fame. The thresholdθ_(TAB) may be changed to any value in accordance with the distributionof the markers or the environment where the markers are placed.

By performing determination using such an evaluation value, an imagecaptured at a viewpoint different from those of all the images detectedto be used can be automatically selected.

Modification 1-4

In the first embodiment, among the images to be used, images captured bythe image pickup unit 100 at the same position and orientation are notexcluded or updated. However, images captured by the image pickup unit100 at the same position and orientation are unnecessary since they haveno information for improving the accuracy of measuring the markerplacement information. Therefore, images captured by the image pickupunit 100 at the same position and orientation should be excluded orupdated. A process of excluding or updating images captured atsubstantially the same position and orientation will now be describedbelow.

Given a vector {right arrow over (C)} extending from the position C_(m)of the image pickup unit 100 at the time the image pickup unit 100 hascaptured the currently obtained image to the barycentric position G_(m)obtained from placement information regarding each marker in the markergroup detected in the obtained image, and vectors {right arrow over(D)}₁, {right arrow over (D)}₂, . . . , and {right arrow over (D)}_(m-1)extending from the positions C₁, C₂, . . . , and C_(m-1) of the imagepickup unit 100 at the times the image pickup unit 100 has capturedimages that have already been determined to be used to the barycentricposition G_(m). Angles θ_(CD1), θ_(CD2), . . . , and θ_(CDm-1) definedby the vector {right arrow over (C)} and the vectors {right arrow over(D)}₁, {right arrow over (D)}₂, . . . , and {right arrow over (D)}_(m-1)are calculated:

$\begin{matrix}\begin{matrix}{{\theta_{CDi} = {\arccos\;\frac{\overset{\rightarrow}{C} \cdot {\overset{\rightarrow}{D}}_{i}}{{\overset{\rightarrow}{C}} \cdot {{\overset{\rightarrow}{D}}_{i}}}}},} & \left( {{i = 1},2,{{\cdots\mspace{14mu} m} - 1}} \right)\end{matrix} & \left( {1\text{-}21} \right)\end{matrix}$

The ratio β_(i) of the vector {right arrow over (C)} to each of thevectors {right arrow over (D)}₁, {right arrow over (D)}₂, . . . , and{right arrow over (D)}_(m-1) is calculated:

$\begin{matrix}\begin{matrix}{{\beta_{i} = \frac{{\overset{\rightarrow}{D}}_{i}}{\overset{\rightarrow}{C}}},} & \left( {{i = 1},2,{{\cdots\mspace{14mu} m} - 1}} \right)\end{matrix} & \left( {1\text{-}22} \right)\end{matrix}$In the case that θ_(CDi) satisfies the following:θ_(TCD)<θ_(CDi)  (1-23)where i=1, 2, . . . , m−1, and β satisfies the following:β_(T1L)≦β_(i)≦β_(T2L)  (1-24)then, it is determined that the currently obtained image is captured atsubstantially the same viewpoint position as that of the image(s) thathave already been determined to be used, and that frame serves as a j-thframe (j=1, 2, . . . , J) where J is the total of images determined tobe captured at the same viewpoint position. In expressions (1-23) and(1-24), θ_(TCD), β_(T1L), and β_(T2L) are preset thresholds.

Next, whether the currently obtained image has been captured with thesame orientation as that of the image(s) that have already beendetermined to be used is determined. The angular difference θ_(mj)between a visual axis vector t_(m) of the image pickup unit 100 at thetime the image pickup unit 100 has captured the obtained image and avisual axis vector t_(j) in the j-th frame determined to be captured atthe same viewpoint position is calculated:

$\begin{matrix}\begin{matrix}{{\theta_{mj} = {\arccos\;\frac{{\overset{\rightarrow}{t}}_{m} \cdot {\overset{\rightarrow}{t}}_{j}}{{{\overset{\rightarrow}{t}}_{m}} \cdot {{\overset{\rightarrow}{t}}_{j}}}}},} & \left( {{j = 1},2,{\cdots\mspace{14mu} J}} \right)\end{matrix} & \left( {1\text{-}25} \right)\end{matrix}$The angular difference θ_(mj) serves as a third evaluation value.

Next, the rotation angular difference around the visual axis iscalculated. Let a vector v_(m) orthogonal to the visual axis vector be avector t_(y)=[0 1 0]^(t) in the y-axis direction in the cameracoordinate system. The vector v_(m) is transformed into the referencecoordinate system using a 3×3 rotation transform matrix R_(m)representing the rough orientation of the image pickup unit 100:v _(m) =R _(m) ·t _(y)  (1-26)

Similarly, let a vector v_(j) orthogonal to the visual axis vector inthe j-th frame determined to have been captured at the same viewpointposition be a vector t_(y)=[0 1 0]^(t) in the y-axis direction in thecamera coordinate system. The vector v_(j) is transformed into thereference coordinate system:v _(j) =R _(j) ·t _(y)  (1-27)

The angular difference between v_(m) and v_(j), that is, the rotationangular difference γ_(mj) around the visual axis, is calculated:

$\begin{matrix}{\gamma_{mj} = {\arccos\;\frac{{\overset{\rightarrow}{v}}_{m} \cdot {\overset{\rightarrow}{v}}_{j}}{{{\overset{\rightarrow}{v}}_{m}} \cdot {{\overset{\rightarrow}{v}}_{j}}}}} & \left( {1\text{-}28} \right)\end{matrix}$The rotation angular difference γ_(mj) (j=1, 2, . . . , J) around thevisual axis serves as a fourth evaluation value.

On the basis of the third and fourth evaluation values obtained in theabove manner, it is determined whether the image has been captured withsubstantially the same orientation, thereby determining whether to usethe image for measurement or not.

In the case that the third evaluation value θ_(mj) satisfies thefollowing:θ_(mj)<θ_(Tmj)(j=1, 2, . . . , N)  (1-29)then, it means that the obtained image has already been captured in thej-th frame from the same viewpoint position and with the sameorientation. Thus, the image will not be used.

In contrast, in the case that the third evaluation value does notsatisfy the condition in expression (1-29), it is then determined on thebasis of the fourth evaluation value whether to use the image.

In the case that the fourth evaluation value γ_(mj) satisfies thefollowing:γ_(mj)>γ_(Tmj)  (1-30)then, the obtained image has been captured from the same viewpointposition and with the same orientation, but the image has been capturedby rotating the image pickup unit 100 around the visual axis. Thus, theimage is determined to be used for measurement. Otherwise, the image isdetermined not to be used. In expression (1-30), γ_(Tmj) is a presetthreshold.

Even in the case of an image determined in the above manner not to beused for measurement since it has been captured from the same viewpointposition and with the same orientation, if the number of markersdetected in the image is large, the image replaces a previous image thathas been determined to be used, thereby updating the image.

Modification 1-5

In document 4, non-linear optimization is performed by placing acondition of constraint on markers using a priori knowledge of themarkers. Since images are automatically obtained in the firstembodiment, a geometric constraint condition cannot be placed on unknownmarkers in calculating the unknown markers. However, in the case that ageometric constraint condition is known in advance, such as in the casethat the detected markers are coplanar, non-linear optimization may beperformed by placing a geometric constraint condition.

Even in the case of unknown markers, non-linear optimization may beperformed without placing a geometric constraint condition. Thereafter,a geometric constraint condition may be added by an operator using aninput device, such as a mouse or a keyboard, and then non-linearoptimization may be performed again.

Modification 1-6

An evaluation value shown below may be used to determine whether animage is to be used for calculating the marker placement information.

Images already determined to be used to calculate an unknown parameterare numbered frame numbers 1, 2, . . . , m−1, and it is then determinedwhether to use an obtained image as an m-th frame.

The obtained image is compared with the (m−1)-th frame. Among detectedmarkers, an image with an overlapping portion greater than or equal tothreshold T % is regarded as an m-th frame candidate.

Next, given a vector {right arrow over (A)} extending from the positionC_(m-1), of the image pickup unit 100 at the time the image pickup unit100 has captured the (m−1)-th frame to the barycentric position G_(m-1)of a marker group detected in the (m−1)-th frame, a vector {right arrowover (B)} extending from the position C_(m) of the image pickup unit 100at the time the image pickup unit 100 has captured the obtained image tothe barycentric position G_(m-1), and a vector {right arrow over (C)}extending from the position C_(m-1) to the position C_(m) of the imagepickup unit 100. If the following expression is satisfied, it isdetermined that the obtained image will be used for calculation:

$\begin{matrix}{\frac{{\overset{\rightarrow}{A}} \cdot {\overset{\rightarrow}{B}}}{\overset{\rightarrow}{C}} < D_{TH}} & \left( {1\text{-}31} \right)\end{matrix}$where D_(th) is a preset threshold.

The accuracy of measurement using images, such as stereographicmeasurement, is affected by the distance to a measurement target and thedistance between image capturing positions. Expression (1-31) takes intoconsideration the distance ∥{right arrow over (A)}∥ between the positionof the image pickup unit 100 at the time the image pickup unit 100 hascaptured the (m−1)-th frame and the barycentric position serving as therepresentative position of the markers, the distance ∥{right arrow over(B)}∥ between the position of the image pickup unit 100 at the time theimage pickup unit 100 has captured the obtained image and thebarycentric position, and the distance ∥{right arrow over (C)}∥ betweenthe image capturing positions. That is, the shorter the distance to themarker representative value, the smaller the ∥{right arrow over(A)}∥·∥{right arrow over (B)}∥. The longer the base line, the greaterthe ∥{right arrow over (C)}∥.

Thus, if expression (1-31) is satisfied, an image necessary forcalculating the marker placement information can be determined.

Modification 1-7

In the first embodiment, in order to take into consideration thedistance to each marker serving as a measurement target, the evaluationvalues are obtained on the basis of the barycentric position of themarkers in the three-dimensional space, and it is then determinedwhether to use that image for calculating the marker placementinformation. In other words, the evaluation values are obtained on thebasis of the vector extending from the position C_(m-1) of the imagepickup unit 100 at the time the image pickup unit 100 has captured the(m−1)-th frame to the barycentric position G_(m-1) of the marker groupdetected in the (m−1)-th frame and the vector extending from theposition C_(m) of the image pickup unit 100 at the time the image pickupunit 100 has captured the obtained image to the barycentric positionG_(m-1).

Instead of obtaining the barycentric position of the marker positions asthe representative position of the markers, the representative positionmay be obtained on the visual axis representing the direction toward thecenter of the image capturing range of the image pickup unit 100. On thebasis of the rough position of each of the detected markers in thethree-dimensional space, the depth of each marker in the cameracoordinate system is obtained, and the average depth of the markers isrepresented by z_(mean). The representative position of the markers inthe camera coordinate system is represented by t_(am)=(0, 0, z_(mean)).Since the rough position and orientation of the image pickup unit 100can be obtained by the image-pickup-unit-position/orientation obtainingunit 1040, the representative position t_(am) of the markers in thecamera coordinate system is transformed into a position in the referencecoordinate system.

The representative position subsequent to the coordinate transform isrepresented by t′_(am). Instead of using the barycentric positionserving as the representative position, which has been used in the firstembodiment, t′_(am) may be used as the representative position.

Second Exemplary Embodiment

In the first embodiment, it is determined whether to use an image forcalculating placement information regarding each marker placed in anenvironment or on an object, and the calculation is performed.

In a second exemplary embodiment, an image to be used to calculate theorientation of an orientation sensor in the camera coordinate system isautomatically determined, and the orientation of the orientation sensormounted on an image pickup unit in the camera coordinate system iscalculated.

The orientation sensor used in the second embodiment mainly includes agyro sensor that detects the angular velocity in the directions of threeaxes and an acceleration sensor that detects the acceleration in thedirections of three axes, and the orientation sensor measures the 3-DOForientation on the basis of a combination of measured values of theangular velocity and the acceleration.

In general, the orientation sensor using the gyro sensor obtains theorientation by integrating angular velocity information output from thegyro sensor. Thus, the obtained orientation contains a drift error.However, by additionally using the acceleration sensor, the direction ofthe earth's gravitational force can be measured, and hence, highlyaccurate tilt angles can be measured.

In contrast, an absolute reference cannot be obtained regarding anazimuth serving as a rotation around the gravity axis, and the drifterror cannot be corrected. Therefore, the measurement accuracy of theazimuth is lower than that of the tilt angles.

FIG. 7 schematically shows an exemplary structure of an informationprocessing apparatus 7000 according to the second embodiment.

An image pickup unit 700 is connected to the information processingapparatus 7000. The image pickup unit 700 captures an image of a spacein which markers are placed and outputs the image to the informationprocessing apparatus 7000. An orientation sensor 705 is mounted on theimage pickup unit 700. At the same time as the image pickup unit 700captures the image, a value measured by the orientation sensor 705 isoutput to the information processing apparatus 7000.

An image obtaining unit 7010 obtains the image from the image pickupunit 700.

A marker extraction unit 7020 extracts markers from the obtained image.A marker management unit 7030 manages, regarding the extracted markers,the identifier k_(n) of each square marker P^(kn), the image coordinatesu^(Pkni) of each vertex p^(kni) of each marker P^(kn), and placementinformation regarding each marker P^(kn) as marker information.

An image-pickup-unit-position/orientation obtaining unit 7040 obtainsthe rough position and orientation of the image pickup unit 700 at thetime the image pickup unit 700 has captured the image.

An image determination unit 7050 determines whether to use the imageinput via the image obtaining unit 7010 to calculate the orientation inthe camera coordinate system of the orientation sensor 705 mounted onthe image pickup unit 700.

At the same time as the image is captured, a sensor-output obtainingunit 7060 obtains the output of the orientation sensor 705.

Using an image determined by the image determination unit 7050 to beused, an orientation-sensor-position/orientation calculator 7070calculates the orientation in the camera coordinate system of theorientation sensor 705 mounted on the image pickup unit 700.

Since the image obtaining unit 7010, the marker extraction unit 7020,the marker management unit 7030, and theimage-pickup-unit-position/orientation obtaining unit 7040 are similarto the image obtaining unit 1010, the marker extraction unit 1020, themarker management unit 1030, and theimage-pickup-unit-position/orientation obtaining unit 1040 described inthe first embodiment, detailed descriptions thereof are omitted to avoidredundancy.

FIG. 8 is a flowchart of a process of calculating placement informationregarding the orientation sensor 705 in the second embodiment.

Since steps S8010 to S8030 are similar to steps S2010 to S2030 in thefirst embodiment, descriptions thereof are omitted to avoid redundancy.

In step S8040, it is determined whether to use an obtained image tocalculate the orientation of the orientation sensor 705 in the cameracoordinate system. Although the determination whether to use theobtained image can be made in a manner similar to that described in thefirst embodiment, in this case, the determination is made on the basisof the orientation of the image pickup unit 700 in the secondembodiment. A detailed description of the method of determining an imageto be used will be given below.

FIG. 9 is a flowchart of a process of determining, in step S8040,whether to use the obtained image to calculate the orientation of theorientation sensor 705 in the camera coordinate system.

Since step S9010 is similar to step S3010 in the first embodiment, adescription thereof is omitted to avoid redundancy.

In step S9020, the marker information managed by the marker managementunit 7030 and the position and orientation of the image pickup unit 700,which are obtained by the image-pickup-unit-position/orientationobtaining unit 7040, are input to the image determination unit 7050.

In step S9030, evaluation values for determining whether to use theobtained image to calculate the orientation in the camera coordinatesystem of the orientation sensor 705 mounted on the image pickup unit700 are calculated by the image determination unit 7050. A method ofcalculating the evaluation values will now be described below.

Images already determined to be used to calculate the orientation of theorientation sensor 705 are numbered frame numbers 1, 2, . . . , m−1, andit is then determined whether to use the obtained image as an m-thframe.

The angular difference θ_(mj) between a visual axis vector t_(m) of theimage pickup unit 700 at the time the image pickup unit 700 has capturedthe obtained image and a visual axis vector t_(j) at the time the imagepickup unit 700 has captured the j-th frame (j=1, 2, . . . m−1) alreadydetermined to be used is computed:

$\begin{matrix}\begin{matrix}{{\theta_{mj} = {\arccos\;\frac{{\overset{\rightarrow}{t}}_{m} \cdot {\overset{\rightarrow}{t}}_{j}}{{{\overset{\rightarrow}{t}}_{m}} \cdot {{\overset{\rightarrow}{t}}_{j}}}}},} & \left( {{j = 1},2,{\cdots\mspace{14mu} J}} \right)\end{matrix} & \left( {2\text{-}1} \right)\end{matrix}$The obtained angular difference θ_(mj) (i=1, 2, . . . J) serves as afirst evaluation value.

Next, the rotation angular difference around the visual axis isobtained.

First, the visual axis vector t_(m) in the reference coordinate systemof the image pickup unit 700 at the time the image pickup unit 700 hascaptured the obtained image is obtained. Given a vector t_(z)=[0 0−1]^(t) in the negative direction of the z-axis in the camera coordinatesystem, the visual axis vector t_(m) is obtained on the basis of therough orientation R_(m) of the image pickup unit 700:t _(m) =R _(m) ·t _(z)  (2-2)

Let a vector v_(m) orthogonal to the visual axis vector t_(m) of theimage pickup unit 700 at the time the image pickup unit 700 has capturedthe obtained image be a vector t_(y)=[0 1 0]^(t) in the y-axis directionin the camera coordinate system. The vector v_(m) in the referencecoordinate system is obtained on the basis of the rough orientationR_(m) of the image pickup unit 700:v _(m) =R _(m) ·t _(y)  (2-3)

Similarly, let a vector v_(j) orthogonal to the visual axis vector inthe j-th frame be a vector t_(y)=[0 1 0]^(t) in the y-axis direction inthe camera coordinate system. The vector v_(j) is transformed bymultiplying t_(y) by the orientation R_(j) of the image pickup unit 700at the time the image pickup unit 700 has captured the j-th frame:v _(j) =R _(j) ·t _(y)  (2-4)

The angular difference between v_(m) and v_(j), that is, the rotationangular difference γ_(mj) around the visual axis, is computed:

$\begin{matrix}{\gamma_{mj} = {\arccos\;\frac{{\overset{\rightarrow}{v}}_{m} \cdot {\overset{\rightarrow}{v}}_{j}}{{{\overset{\rightarrow}{v}}_{m}} \cdot {{\overset{\rightarrow}{v}}_{j}}}}} & \left( {2\text{-}5} \right)\end{matrix}$The rotation angular difference γ_(mj) (j=1, 2, . . . , J) around thevisual axis serves as a second evaluation value.

In step S9040, it is determined on the basis of the obtained evaluationvalues whether to use the obtained image for calculation.

In the case that, for all images (j=1, 2, . . . J) determined to beused, the first evaluation value θ_(mj) satisfies the following:θ_(mj)<θ_(Tmj)(j=1, 2, . . . ,N)  (2-6)then, it means that the obtained image has been captured from thedirection of a viewpoint at which no image has been captured yet, and itis thus determined that the image will be used. In expression (2-6),θ_(Tmj) is a preset threshold.

In the case that expression (2-6) is not satisfied, there is aprobability that the obtained image is captured with the sameorientation. On the basis of the second evaluation value, it isdetermined whether to use the obtained image.

In the case that the second evaluation value γmj satisfies thefollowing:γ_(mj)>γ_(Tmj)  (2-7)then, the obtained image has been captured from the same viewpointdirection, but the image has been captured by rotating the image pickupunit 700 around the visual axis. Thus, the image is determined to beused for calculation. In expression (2-7), γ_(Tmj) is a presetthreshold.

In the case that expression (2-7) is not satisfied, it is determined notto use the obtained image to calculate the orientation of theorientation sensor 705.

In step S9050, the result of determination obtained in step S9040showing whether to use the image is output.

Referring back to FIG. 8, in step S8050, in the case that the image isdetermined in step S8040 to be used to calculate the orientation of theorientation sensor 705 and that the number m of images determined to beused is greater than or equal to two (m≧2), the flow proceeds to stepS8060. In contrast, in the case that the image is determined in stepS8040 not to be used, the flow returns to step S8010.

Next step S8060 in which the orientation of the orientation sensor 705is calculated will be described in detail.

In step S8060, the orientation ω_(cs) in the camera coordinate system ofthe orientation sensor 705 mounted on the image pickup unit 700 iscalculated using images captured at a plurality of viewpoint positions,which are determined by the image determination unit 7050 to be used.

The sensor-output obtaining unit 7060 obtains the sensor output value ofthe orientation sensor 705. The position of the image pickup unit 700,an azimuth drift correction value φ_(τ), and ω_(cs) are obtained bynon-linear optimization so as to minimize the error between the detectedposition of each marker in the image in the reference coordinate systemand the calculated position of the marker in the image, which iscalculated on the basis of the position of the image pickup unit 700,the sensor measurement value, the azimuth drift correction value, andω_(cs).

Given a sensor output value R_(wsτ) of the orientation sensor 705 in aworld coordinate system, a rotation matrix ΔR(φ_(τ)) (azimuth drifterror correction value) for rotating the image pickup unit 700 by φ_(τ)in the azimuth direction (around the gravity axis), and a 3×3 rotationmatrix R(ω_(sc)) determined by ω_(cs). A transform equation fortransforming the reference coordinate system to the camera coordinatesystem is derived:R _(WC) _(τ) =ΔR(φ_(τ))·R _(WS) _(τ) ·R(ω_(SC))  (2-8)

The orientation ω_(cs) of the orientation sensor 705 in the cameracoordinate system is processed as a three-element vector ω_(cs)=[ξ ψξ]^(T). The orientation ω_(cs) of the orientation sensor 705 in thecamera coordinate system, the position t_(WCτ)=[x_(tτ) y_(tτ)z_(tτ)]^(T) of the image pickup unit 700 at a certain viewpoint position(represented by an identifier τ), and an “azimuth drift error correctionvalue” φ_(τ) of the sensor measurement value at the time the image hasbeen captured are unknown. These unknown parameters are expressed as a3+4L-dimensional state vector:s[ω _(SC) ^(T) t _(WC1) ^(T)φ₁ . . . t _(WCτ) ^(T)φ_(τ) . . . t _(WCL)^(T)φ_(L)]^(T)where L is the total of images captured at different viewpoints.

An appropriate initial value is given to the state vector s. The initialvalue of the position t_(WCτ) of the image pickup unit 700 can beobtained by the image-pickup-unit-position/orientation obtaining unit7040. The initial value of ω_(cs) can be obtained by a known method inwhich each parameter value representing ω_(cs) is interactivelyincreased/decreased to adjust the value by trial and error. Morespecifically, a virtual object can be projected onto the image on thebasis of the initial value of the position of the image pickup unit 700,ω_(cs), and the sensor output value. Thus, an operator adjusts theposition using an input device, such as a keyboard, such that thecoordinates of the projected virtual object are accurately registeredwith a corresponding real object. A geomagnetic sensor may be used toobtain the rough azimuth.

Another method of obtaining the initial value of ω_(cs) is disclosed inJapanese Patent Laid-Open No. 2003-203252 (corresponding to U.S. Pat.No. 7,095,424). A virtual object based on a preset orientation isdisplayed on a display. Next, an operator monitors and adjusts theorientation of an image pickup device such that the displayed virtualobject has a correct positional relationship with a real space. Finally,the orientation of an orientation sensor in the reference coordinatesystem and the orientation of the orientation sensor in the cameracoordinate system are calculated in accordance with an output of theorientation sensor at that time.

The initial value of ω_(τi) is calculated by a method described inJapanese Patent Laid-Open No. 2005-107248 (corresponding to U.S. PatentApplication No. 2005/0068293) using the initial value of ω_(cs).

Marker information similar to that managed by the marker management unit1030 described in the first embodiment is managed by the markermanagement unit 7030. Theoretical values of the image coordinatesu=[u_(x), u_(y)]T of the vertices of all markers are calculated on thebasis of the image coordinates of each vertex p^(ki) of eachquadrangular marker P^(k), placement information and identifier of eachquadrangular marker P^(k), and the state vector s. In this case, thetheoretical values of the image coordinates refer to the coordinatesthat should be observed in a captured image of the markers, which arecalculated on the basis of the position and orientation of the imagepickup unit 700 in the reference coordinate system and placementinformation regarding the markers in the reference coordinate system.

The orientation of the image pickup unit 700 in the reference coordinatesystem can be calculated on the basis of a coordinate transform from theorientation of the orientation sensor 705 in a sensor coordinate systemto the orientation in the reference coordinate system, the orientationof the orientation sensor 705 in the sensor coordinate system (measuredvalue of the orientation of the orientation sensor 705), the azimuthdrift error correction value, and a coordinate transform (ω_(sc)) fromthe orientation of the orientation sensor 705 to the orientation of theimage pickup unit 700. Calculation of a theoretical estimate of acertain marker u=[u_(x), u_(y)]^(T) is expressed using the state vectors:u=F(s)  (2-9)where F is a function including a viewing transform from the referencecoordinate system to the camera coordinate system and a perspectiveprojection transform.

Regarding the vertices of all the markers, an error Δu between theactual image coordinates of each marker v=[v_(x), v_(y)]^(T) andcorresponding theoretical values of the image coordinates u=[u_(x),u_(y)]^(T) is calculated:Δu=v−u  (2-10)

The state vector s is optimized so as to minimize Δu. Using a vector Δsfor correcting the state vector s, Δu can be expressed in the followingmanner by performing first-order approximation through Taylor expansion:

$\begin{matrix}{{\Delta\; u} \approx {\left\lbrack \frac{\partial u}{\partial s} \right\rbrack\Delta\; s}} & \left( {2\text{-}11} \right)\end{matrix}$In this case,

$J_{us} = \left\lbrack \frac{\partial u}{\partial s} \right\rbrack$where J_(us) is a 2×(3+4L) Jacobian matrix having, as each element, apartial differential coefficient obtained by partially differentiating uin equation (2-10) with respect to the state vector s.

Expression (2-11) is a relational expression regarding one of thevertices of L images to be used to calculate the orientation of theorientation sensor 705.

Given n markers extracted from the images determined to be used forcalculation. The following expression holds true:

$\begin{matrix}{\begin{bmatrix}{\Delta\; u_{1}} \\{\Delta\; u_{2}} \\\vdots \\{\Delta\; u_{n}}\end{bmatrix} \approx {\begin{bmatrix}J_{{us}\; 1} \\J_{{us}\; 2} \\\vdots \\J_{usn}\end{bmatrix}\Delta\; s}} & \left( {2\text{-}12} \right)\end{matrix}$The unknown parameter Δs to be calculated can be calculated by aleast-squares method. Since this can be solved by a manner similar tothat in the first embodiment, a repeated description thereof is omitted.

Since Δs is a (3+4L)-dimensional vector, Δs can be obtained by detectingat least one square marker in two captured images.

Using the corrected value Δs, the state vector s is corrected to obtaina new s:s+Δs→s  (2-13)

Using a determination reference, such as whether ΣΔs is less than orequal to a preset threshold, or whether the corrected value As is lessthan or equal to a preset threshold, it is determined whether thecalculation has converged. If the calculation has not converged, thecorrected state vector s serves as an initial value, and the calculationis repeated.

If it is determined that the calculation has converged, ω_(sc) includedin the obtained state vector s is output. As a parameter indicating thelatest azimuth drift error correction value, φτ can be additionallyoutput.

Finally in step S8070, it is determined whether to end the calculation.When the operator gives an instruction to end the calculation, the flowends. When the operator gives an instruction to continue the calculation(recalibration), an image is obtained again.

In the case that the virtual object rendered in the image using theobtained ω_(sc) and φτ is correctly registered with the real object, theoperator ends the calculation. Otherwise, the operator determines tocontinue the calculation.

In the above manner, images appropriate to calculate the orientation ofthe orientation sensor 705, which is mounted on the image pickup unit700, with respect to the image pickup unit 700 are automaticallyobtained, and placement information regarding the orientation sensor 705can be highly accurately measured without depending on the user'sknowledge and skills.

Third Exemplary Embodiment

In the first embodiment, the marker placement information or theposition and orientation of the image pickup unit 100 are obtained as ameasurement target. In the second embodiment, the orientation of the3-DOF orientation sensor 705 in the camera coordinate system is obtainedas a measurement target.

In a third exemplary embodiment, the position and orientation of asensor coordinate system defined by a 6-DOF position/orientation sensorrelative to the reference coordinate system and the position andorientation of the 6-DOF position/orientation sensor mounted on an imagepickup unit relative to the image pickup unit are obtained as unknownparameters. In the third embodiment, the same markers as those describedin the first and second embodiments are used.

FIG. 10 is a flowchart of a process of calculating placement informationregarding the 6-DOF position/orientation sensor according to the thirdembodiment.

Since steps S210 to S230 are similar to steps S2010 to S2030 in thefirst embodiment, descriptions thereof are omitted to avoid redundancy.

In step S240, an image determination unit determines whether to use anobtained image to calculate the unknown parameters. Since thedetermination method is similar to that described in the secondembodiment, a description thereof is omitted to avoid redundancy.Alternatively, whether to use the image to calculate the unknownparameters may be determined in a manner similar to that described inthe first embodiment.

In step S260, the unknown parameters are calculated.

In the third embodiment, the unknown parameters include a parameters_(WT) representing the position and orientation of the sensorcoordinate system relative to the reference coordinate system and aparameter s_(LT) representing the position and orientation of a receivercoordinate system of the 6-DOF sensor relative to the camera coordinatesystem.

Given the position of the sensor coordinate system in the referencecoordinate system t_(WT)=[t_(xWT) t_(yWT) t_(zWT)]^(T) and theorientation of the sensor coordinate system relative to the referencecoordinate system ω^(WT)=[ω^(xWT) ω^(yWT) ω_(zWT)]^(T). Then,S_(WT)=[t_(xWT) t_(yWT) t_(zWT) ω_(xWT) ω_(yWT) ω_(zWT)]^(T). Given theposition of the receiver coordinate system of the 6-DOF sensor in thecamera coordinate system t_(LT)=[t_(xLT) t_(yLT) t_(zLT)]^(T) and theorientation of the receiver coordinate system of the 6-DOF sensorrelative to the camera coordinate system ω_(LT)=[ω_(xLT) ω_(yLT)ω_(zLT)]^(T). Then, s_(LT)=[t_(xLT) t_(yLT) t_(zLT) ω^(xLT) ω_(yLT)ω_(zLT)]^(T). Thus, the 12 unknown parameters are expressed as a statevector s:s[t_(xWT)t_(yWT)t_(zWT)ω_(xWT)ω_(yWT)ω_(zWT)t_(xLT)t_(yLT)t_(zLT)ω_(xLT)ω_(yLT)ω_(zLT)]^(T)Appropriate initial values are given to the state vector s. For example,rough values measured manually by the user are input in advance.

Theoretical values of the image coordinates u=[u_(x), u_(y)]T of thevertices of all markers are calculated on the basis of the imagecoordinates of each vertex p^(ki) of each quadrangular marker P^(k), asin the first embodiment, placement information and identifier of eachquadrangular marker P^(k), and the state vector s. In this case, given athree-dimensional vector t representing the position of the origin of athree-dimensional coordinate system A relative to a certainthree-dimensional coordinate system B, and a 3×3 rotation matrix Rrepresenting the orientation, then, the coordinates x_(B)(three-dimensional vector) in the coordinate system B of a pointrepresented by x_(A) (three-dimensional vector) representing theposition in the coordinate system A are expressed:

$\begin{matrix}{\begin{bmatrix}x_{B} \\1\end{bmatrix} = {\begin{bmatrix}R & t \\0 & 1\end{bmatrix}\begin{bmatrix}x_{A} \\1\end{bmatrix}}} & \left( {3\text{-}1} \right)\end{matrix}$Suppose M collectively denotes R and t, and the following is derived:

$\begin{matrix}{M = \begin{bmatrix}R & t \\0 & 1\end{bmatrix}} & \left( {3\text{-}2} \right)\end{matrix}$

FIG. 11 illustrates a transform of a marker placed in the referencecoordinate system to the camera coordinate system.

Let M_(WT) be the position and orientation s_(WT) of the sensorcoordinate system of the 6-DOF position/orientation sensor in thereference coordinate system, M_(TS) be the output value of the 6-DOFposition/orientation sensor, M_(CS) be the position and orientations_(LT) in the camera coordinate system of the 6-DOF position/orientationsensor mounted on the image pickup unit, and M_(WM) be placementinformation regarding the marker in the reference coordinate system.

A matrix M_(CM) for transforming the marker placed in the referencecoordinate system to the camera coordinate system is:M _(CM) =M _(CS)·(M _(TS))⁻¹·(M _(wT))⁻¹ ·M _(wM)  (3-3)

Calculation of a theoretical estimate of a certain marker u=[u_(x),u_(y)]^(T) is expressed using the state vector s as:u=F(s)  (3-4)where F is a function including a viewing transform from the referencecoordinate system to the camera coordinate system and a perspectiveprojection transform.

A parameter that minimizes an error Δu between the actual imagecoordinates of the marker v=[v_(x), v_(y)]^(T) and correspondingtheoretical values of the image coordinates u=[u_(x), u_(y)]^(T) iscalculated.

Using a vector Δs for correcting the state vector s, Δu can be expressedin the following manner by performing first-order approximation throughTaylor expansion:

$\begin{matrix}{{{\Delta\; u} \approx {\left\lbrack \frac{\partial u}{\partial s} \right\rbrack\Delta\; s}}{{where}\mspace{14mu}\left\lbrack \frac{\partial u}{\partial s} \right\rbrack}} & \left( {3\text{-}5} \right)\end{matrix}$is a Jacobian matrix. Since the Jacobian matrix can be calculated by aknown method, a description of a specific method of calculating theJacobian matrix is omitted.

Given a Jacobian matrix J_(us1) regarding a point in a certain image.Also, given n markers extracted from an image determined to be used tocalculate the unknown parameter in expression (3-5). Then, the followingexpression is derived:

$\begin{matrix}{\begin{bmatrix}{\Delta\; u_{1}} \\{\Delta\; u_{2}} \\\vdots \\{\Delta\; u_{n}}\end{bmatrix} \approx {\begin{bmatrix}J_{{us}\; 1} \\J_{{us}\; 2} \\\vdots \\J_{usn}\end{bmatrix}\Delta\; s}} & \left( {3\text{-}6} \right)\end{matrix}$The unknown parameter Δs to be obtained can be calculated by aleast-squares method. Since this can be solved by a manner similar tothose in the first and second embodiments, a repeated descriptionthereof is omitted.

Using the corrected value Δs, the state vector s is corrected to obtaina new s:s+Δs→s  (3-7)

Using a determination reference, such as whether the error vector Δs isless than or equal to a preset threshold, or whether the corrected valueΔs is less than or equal to a preset threshold, it is determined whetherthe calculation has converged. If the calculation has not converged, thecorrected state vector s serves as an initial value, and the calculationis repeated.

If it is determined that the calculation has converged, the coordinatetransform (position and orientation) of the sensor coordinate system ofthe 6-DOF sensor mounted on the image pickup unit in the referencecoordinate system and the coordinate transform (position andorientation) of the receiver coordinate system of the 6-DOF sensor inthe camera coordinate system are output.

In step S270, it is determined whether to end the calculation. When theoperator gives an instruction to end the calculation, the flow ends.When the operator gives an instruction to continue the calculation(recalibration), an image is obtained again.

In the case that the virtual object rendered in the image using theobtained state vector s is correctly registered with the real object,the operator ends the calculation. Otherwise, the operator determines tocontinue the calculation.

In the above manner, highly accurate measurement can be performedwithout depending on the user's knowledge and skills by automaticallydetermining and calculating images necessary to calculate the positionand orientation of the sensor coordinate system defined by the 6-DOFposition/orientation sensor relative to the reference coordinate systemof the 6-DOF sensor, and the position and orientation of the receivercoordinate system of the 6-DOF sensor in the camera coordinate system.

Modification 3-1

In the first to third embodiments, the Newton-Raphson method has beenused, which is an algorithm for finding the optimal solution byrepeating a process of Taylor-expanding a non-linear function inoptimization calculation and linearizing the result by first-orderapproximation to obtain a corrected value.

However, the corrected value may not necessarily be calculated by theNewton-Raphson method. For example, the corrected value may be obtainedby the Levenberg-Marquardt method, which is a known iterative non-linearequation algorithm, or by a steepest descent method.

Alternatively, a robust estimation method, such as M estimation, whichcan stabilize the calculation by obtaining the solution while reducingeffects of statistical outliers, may be employed.

Fourth Exemplary Embodiment

In the first to third embodiments, the methods of automaticallydetermining images that are necessary to calculate unknown parametersusing captured images of markers have been described.

In a fourth exemplary embodiment, the measurement is simplified bypresenting to the user, through a display, an area or a path for movingthe image pickup unit. In the first embodiment, it is determined whetherto use an input image to calculate placement information regarding eachmarker.

In contrast, in the fourth embodiment, an image capturing area or animage capturing position and orientation determined to be used tocalculate the marker placement information are calculated, and theposition and orientation are presented to the user, thereby improvingthe input image itself.

FIG. 12 schematically shows an exemplary structure of an informationprocessing apparatus 300 according to the third embodiment.

An image pickup unit 305 and a display 365 are connected to theinformation processing apparatus 300.

An image obtaining unit 310 obtains an image from the image pickup unit305.

A marker extraction unit 320 extracts markers from the obtained image.

A marker management unit 330 manages, regarding the extracted markers,the identifier k_(n) of each quadrangular marker P^(kn), the imagecoordinates u^(Pkni) of each vertex p^(kni) of each marker P^(kn), andplacement information regarding each marker P^(kn) as markerinformation.

An image-pickup-unit-position/orientation obtaining unit 340 obtains therough position and orientation of the image pickup unit 305 at the timethe image pickup unit 100 has captured the image.

An image-capturing-area calculator 350 calculates an area in which animage necessary for satisfactory measurement can be captured.

An image-capturing-area presentation unit 360 displays the areacalculated by the image-capturing-area calculator 350 on the display365, thereby presenting the area to the user.

An image determination unit 370 determines whether to use the obtainedimage to calculate placement information regarding each marker.

An unknown parameter calculator 380 calculates the marker placementinformation as an unknown parameter.

Since the image obtaining unit 310, the marker extraction unit 320, themarker management unit 330, and theimage-pickup-unit-position/orientation obtaining unit 340 are similar tothe image obtaining unit 1010, the marker extraction unit 1020, themarker management unit 1030, and theimage-pickup-unit-position/orientation obtaining unit 1040 described inthe first embodiment, descriptions thereof are omitted to avoidredundancy.

FIG. 13 is a flowchart of a process according to the fourth embodiment.

Since steps S410 to S430 are similar to steps S2010 to S2030 in thefirst embodiment, descriptions thereof are omitted to avoid redundancy.

In step S440, an image capturing area is calculated. A method ofcalculating the image capturing area is described in detail using FIG.14.

FIG. 14 illustrates the method of calculating the image capturing area.

Images already determined to be used are numbered frame numbers 1, 2, .. . , m−1. In addition, given a vector {right arrow over (A)} extendingfrom the position C_(m-1) of the image pickup unit 305 at the time theimage pickup unit 305 has captured the (m−1)-th frame to the barycentricposition G_(m-1) of a marker group extracted from the (m−1)-th frame.Suppose that the barycentric position G_(m-1) is a representativeposition of the target markers to be calculated. Next, given a vector{right arrow over (B)} extending from the barycentric position G_(m-1)subtending a threshold angle θ_(TAB) relative to the vector {right arrowover (A)}. Assume an area V that expands infinitely, which is formed byrotating {right arrow over (B)} around {right arrow over (A)} serving asa rotation axis. Let V be an area other than the area V. In the casethat the image capturing position resides in the area V, it means thatthe image has been captured at a viewpoint differing from that of the(m−1)-th frame, meaning that this is an area in which an image with aparallax that is sufficiently large to calculate the unknown parameterscan be captured. Thus, the image capturing in the area V serves as thenext image capturing area in which an image should be captured next.

Given vectors {right arrow over (B)}₁ and {right arrow over (B)}₂ withsize ratios a₁ and a₂ relative to {right arrow over (A)}, respectively,which can be obtained as:|{right arrow over (B)} ₁|=α₁ |{right arrow over (A)}||{right arrow over (B)} ₂|=α₂ |{right arrow over (A)}|  (4-1)where a₁<a₂, and hence:|{right arrow over (B)} ₁ |<|{right arrow over (B)} ₂|  (4-2)

The area V is an area that expands infinitely. Alternatively, it is alsomeaningful to limit the area because, when an image is captured at aposition closer to the representative position of the measurementtargets than the viewpoint position at which the (m−1)-th frame has beencaptured, the accuracy of detecting the markers in the image isimproved, thereby improving the measurement accuracy of the unknownparameters.

In contrast, in the case that an image is captured at a position fartheraway from the representative position of the measurement targets thanthe viewpoint position at which the (m−1)-th frame has been captured,the accuracy of detecting the markers in the image is degraded. However,the image capturing area is expanded, giving a possibility of detectinga new marker. Therefore, the area V is updated in the following manner.

First, assume an area V₁ is obtained by rotating {right arrow over (B)}₁around {right arrow over (A)} serving as a rotation axis, and an area V₂is obtained by rotating {right arrow over (B)}₂ around {right arrow over(A)} serving as a rotation axis. The area V₂−V₁ is updated as the areaV. Assume an area V is an area other than the area V. In the case thatthe image capturing position resides in the area V, it means that thisserves as the next image capturing area in which an image should becaptured next. From this point onward, V serves as the image capturingarea.

Referring back to FIG. 13, in step S450, the image capturing area ispresented to the user on the basis of the image capturing areacalculated in step S440.

A method of presenting the image capturing area to the user will bedescribed now. FIG. 15A illustrates an example in which the imagecapturing area is presented to the user.

The position and orientation of the image pickup unit 305 at the timethe image pickup unit 305 has captured the obtained image have alreadybeen obtained by the image-pickup-unit-position/orientation obtainingunit 340. As shown in FIG. 15A, it is presented to the user using textand arrows to move the image pickup unit 305 to the nearest imagecapturing area V.

The user watches and recognizes the presentation of this virtual imagecapturable area or the image capturing position and orientation of avirtual image pickup unit and confirms the position in the real space atwhich an image should be captured.

In the case that images sufficient for calculating the unknown parametercan be obtained because of the presentation in this manner, the unknownparameter can be calculated by the method described in the firstembodiment.

Modification 4-1

In the fourth embodiment, an image capturing area to be captured next iscalculated, and text and arrows prompting the user to move the imagepickup unit 305 is presented to the user via the display 365. However,the presentation is not limited to the prompting text and arrows.

For example, the image capturing area itself shown in FIG. 14 or thenext image capturing position and orientation of the image pickup unit305 may be presented by rendering and superposing virtual CG images. Aslong as the next image capturing position and orientation or an imagecapturable area can be presented, the presentation method may employpattern images, three-dimensional CG images displayed in lines, andcharacters.

Although the user is prompted to move the image pickup unit 305 to thenearest image capturing area V in the fourth embodiment, a motion vectorfrom the (m−2)-th frame to the (m−1)-th frame may be calculated, and theuser may be prompted to move the image pickup unit 305 to an imagecapturing area V close to the direction indicated by the motion vector.

Fifth Exemplary Embodiment

In the second embodiment, the method of automatically determining imagesnecessary to calculate unknown parameters using captured images ofmarkers has been described.

In a fifth exemplary embodiment, the measurement is simplified bypresenting to the user, through a display, an area or a path for movingthe image pickup unit. In the second embodiment, it is determinedwhether to use an image to calculate the orientation of the orientationsensor 705 mounted on the image pickup unit 700 in the camera coordinatesystem. In contrast, in the fifth embodiment, an image capturing area oran image capturing position and orientation determined to be used tocalculate the orientation of the orientation sensor mounted on the imagepickup unit in the camera coordinate system, which serves as an unknownparameter, is calculated, and the calculated result is presented to theuser, thereby improving the input image itself.

FIG. 16 schematically shows an exemplary structure of an informationprocessing apparatus 500 according to the fifth embodiment.

An image pickup unit 505 and a display 565 are connected to theinformation processing apparatus 500.

An image obtaining unit 510 obtains an image from the image pickup unit505.

A marker extraction unit 520 extracts markers from the obtained image.

A marker management unit 530 manages, regarding the extracted markers,the identifier k_(n) of each quadrangular marker P^(kn), the imagecoordinates u^(Pkni) of each vertex p^(kni) of marker P^(kn), andplacement information regarding each marker P^(kn) as markerinformation.

An image-pickup-unit-position/orientation obtaining unit 540 obtains therough position and orientation of the image pickup unit 505 at the timethe image pickup unit 505 has captured the image.

An image-capturing-area calculator 550 calculates an area in which animage necessary for satisfactory measurement can be captured.

An image-capturing-area presentation unit 560 displays the areacalculated by the image-capturing-area calculator 550 on the display565, thereby presenting the area to the user.

An image determination unit 570 determines whether to use the obtainedimage to calculate the unknown parameter.

An unknown parameter calculator 580 calculates the orientation of anorientation sensor mounted on the image pickup unit 505 as the unknownparameter.

Since the image obtaining unit 510, the marker extraction unit 520, themarker management unit 530, and theimage-pickup-unit-position/orientation obtaining unit 540 are similar tothe image obtaining unit 7010, the marker extraction unit 7020, themarker management unit 7030, and theimage-pickup-unit-position/orientation obtaining unit 7040 described inthe second embodiment, descriptions thereof are omitted to avoidredundancy.

FIG. 17 is a flowchart of a process according to the fifth embodiment.

Since steps S610 to S630 are similar to steps S8010 to S8030 in thesecond embodiment, descriptions thereof are omitted to avoid redundancy.

In step S640, an image capturing area is calculated. A method ofcalculating the image capturing area is described in detail below.

Images already determined to be used are numbered frame numbers 1, 2, .. . , m−1. In addition, given a visual axis vector t_(j) at the time thej-th frame (j=1, 2, . . . m−1), which has already been determined to beused for calculation, has been captured. Assume a vector {right arrowover (A)}_(i) is obtained by translating t_(j) such that the viewpointposition of t_(j) passes through the origin A₀ of an arbitrarycoordinate system A′. Assume a vector {right arrow over (B)}_(j) has anangular difference θ_(Tmj) relative to {right arrow over (A)}_(j)regarding the origin A₀ of the coordinate system A′, where the angulardifference θ_(Tmj) is a preset threshold. Then, assume an area V_(j) isobtained by rotating {right arrow over (B)}_(j) around {right arrow over(A)}_(j) serving as a rotation axis, and assume an area V _(j), which isan area other than the area V. In the case that the visual axis vectorof the image pickup unit 505 translated in the coordinate system A′resides in the area V _(j), it means that an image can be captured at aviewpoint differing from that of the j-th frame. Thus, regarding all thej-th frames where j=1, 2, . . . , m−1, an image capturing area in whichthe visual axis vector is contained in the area V _(j) serves as thenext image capturing area in which an image should be captured next.

However, even in the case that the visual axis vector is contained inthe area V_(j), if the image pickup unit 505 is rotated around thevisual axis, an image can be captured with a different orientation.Thus, this also serves as an image capturing area.

Among the frames where j=1, 2, . . . , m−1, images in which the visualaxis vector is contained in the area V_(j) are all selected. Among theselected K images, a vector v_(k) orthogonal to the visual axis vectorof a k-th frame (k=1, 2, . . . , K) is assumed as a vector t_(y)=[0 10]^(t) in the y-axis direction in the camera coordinate system. From theorientation R_(k) of the image pickup unit 505 at the time the imagepickup unit 505 has captured the k-th frame, the following is derived:V _(K) =R _(j) ·T _(y)  (5-1)

In the case that the orientation of the image pickup unit 505 has avisual-axis orthogonal vector v_(m) with an angular difference greaterthan or equal to Y_(Tmk) with respect to v_(k) around the visual axisvector, it means that the image pickup unit 505 is rotated around thevisual axis, and hence an image will be captured with a differentorientation. Thus, this serves as the next image capturing area in whichan image should be captured next.

Assume an area W_(k) has a visual-axis orthogonal vector with an angulardifference that is less than or equal to Y_(Tmk), around the visual axisvector, with respect to the vector v_(k) orthogonal to the visual axisvector, and assume an area W _(k) which is an area other than the areaW_(k). The image capturing position/orientation in the case that thevisual axis vector resides in the area V_(j) and the vector v_(m)orthogonal to the visual axis vector resides in the area W _(k) servesas the next image capturing area in which an image should be capturednext.

In step S650, the image capturing area is presented to the user on thebasis of the image capturing area calculated in step S640.

Since a method of presenting the image capturing area to the user issimilar to that in the fourth embodiment, a description thereof isomitted to avoid redundancy.

The user watches and recognizes the presentation of this virtual imagecapturable area or the image capturing position and orientation of avirtual image pickup unit and confirms the position in the real space atwhich an image should be captured.

In the case that images sufficient for calculating the unknown parametercan be obtained because of the presentation in this manner, the unknownparameter can be calculated by the method described in the secondembodiment.

First Modification 5-1

In the fifth embodiment, the unknown parameter is the orientation of theorientation sensor mounted on the image pickup unit 505, and thecapturing area of an image necessary to calculate the unknown parameteris presented. In the case that a 6-DOF sensor serves as a measurementtarget, a similar image-capturing-area calculation method can beemployed to present an image capturing area to the user. As has beendescribed in the third embodiment, the position and orientation of areceiver of the 6-DOF sensor mounted on the image pickup unit and theposition and orientation of the 6-DOF sensor in the reference coordinatesystem can be calculated as unknown parameters.

Modification 5-2

In the first to fourth embodiments, images necessary for calculating theunknown parameters are automatically obtained. However, in the case thatthe operator determines the necessity of obtaining images, images may beobtained manually via an input device, such as a keyboard.

Sixth Exemplary Embodiment

An information processing apparatus according to a sixth exemplaryembodiment enables a user to experience the MR while images necessaryfor calculating the unknown parameters described in the first to fifthembodiments are obtained online and the calculation results arereflected in the MR.

The information processing apparatus according to the sixth embodimentemploys a video see-through head mounted display (HMD) to presentcapturing areas of images needed to calculate the unknown parameters.

The rough position and orientation of the image pickup unit at the timethe image pickup unit has captured the obtained image are obtained bythe image-pickup-unit-position/orientation obtaining unit described inthe first to fifth embodiments. On the basis of the obtained roughposition and orientation of the image pickup unit and the known internalparameters of the image pickup unit, a virtual object can be renderedand superposed on the obtained image.

In the case that the unknown parameters include the internal parametersof the image pickup unit, a virtual object is rendered and superposedusing rough values of the internal parameters of the image pickup unitand the obtained rough position and orientation of the image pickupunit. Using a plurality of images determined to be used to calculate theunknown parameters, the other unknown parameters in the first to fourthembodiments are calculated, and then the unknown parameters that havebeen set as the rough values are calculated correctly. Accordingly, thevirtual object can be correctly superposed on the real world.

Modification 6-1

In the sixth embodiment, the capturing areas of images necessary tocalculate the unknown parameters are presented on the video see-throughHMD.

However, a display for presenting information to the user is not limitedto the video see-through HMD. For example, a cathode-ray tube (CRT) or aliquid crystal display (LCD) may be used, or an optical see-through HMDmay be used.

Modification 6-2

The calculation of the unknown parameters in the first to fifthembodiments may be done separately from general calculation for enablingthe user to experience the MR.

The former involves online determination of images necessary tocalculate the unknown parameters in the first to fifth embodiments andcalculation of the unknown parameters. The latter involves, to enablethe user to experience the MR, general calculation of the position andorientation of the image pickup unit and CG rendering calculation.

The above calculations are divided into a plurality of threads, and thecalculations are performed by a computer having a plurality of centralprocessing unit (CPU) cores referred to as “multicores”. Accordingly,the unknown parameters with relatively high calculation costs can becalculated, while the user is enabled to experience the MR in real time.

Seventh Exemplary Embodiment

In the second embodiment, for the purpose of calculating the orientationof the orientation sensor 705 mounted on the image pickup unit 700 inthe camera coordinate system, images used to calculate the orientationare automatically determined on the basis of the position andorientation of the image pickup unit 700, which have been obtained byobserving the markers.

In a seventh exemplary embodiment, images used to calculate theorientation of the orientation sensor in the camera coordinate systemare automatically determined on the basis of the output value of theorientation sensor.

FIG. 18 schematically shows an exemplary structure of an informationprocessing apparatus 10000 according to the seventh embodiment.

An image pickup unit 11000 is connected to the information processingapparatus 10000. The image pickup unit 11000 outputs an image of a spacein which markers are placed to the information processing apparatus10000. An orientation sensor 12000 is mounted on the image pickup unit11000. At the same time as the image pickup unit 11000 captures theimage, a value measured by the orientation sensor 12000 is output to theinformation processing apparatus 10000.

An image obtaining unit 10010 obtains the image from the image pickupunit 11000.

A marker extraction unit 10020 extracts the markers from the obtainedimage. A marker management unit 10030 manages, regarding the extractedmarkers, the identifier k_(n) of each square marker P^(kn), the imagecoordinates u^(Pkni) of each vertex p^(kni) of each marker P^(kn), andplacement information regarding each marker P^(kn) as markerinformation.

A sensor-output obtaining unit 10040 obtains the sensor output value atthe same time as the image is captured.

An image determination unit 10050 determines whether to use the imageinput via the image obtaining unit 10010 to calculate the orientation ofthe orientation sensor 12000 in the camera coordinate system.

Using images determined by the image determination unit 10050 to beused, an orientation-sensor-position/orientation calculator 10060calculates the orientation of the orientation sensor 12000 in the cameracoordinate system.

Since the image obtaining unit 10010, the marker extraction unit 10020,the marker management unit 10030, and the sensor-output obtaining unit10040 are similar to the image obtaining unit 7010, the markerextraction unit 7020, the marker management unit 7030, and the sensoroutput obtaining unit 7060 described in the second embodiment, detaileddescriptions thereof are omitted to avoid redundancy.

FIG. 19 is a flowchart of a process of calculating placement informationregarding the orientation sensor 12000 in the seventh embodimentrelative to the camera coordinate system.

Since steps S10110 and S10120 are similar to steps S8010 to S8020 in thesecond embodiment, descriptions thereof are omitted to avoid redundancy.

In step S10130, the sensor-output obtaining unit 10040 obtains thesensor output value from the orientation sensor 12000.

In step S10140, the image determination unit 10050 determines whether touse the obtained image to calculate the orientation of the orientationsensor 12000 in the camera coordinate system on the basis of the sensoroutput value input in step S10130. The processing in step S10140 will bedescribed in detail below.

FIG. 20 is a flowchart of a process of determining, in step S10140,whether to use the obtained image to calculate the orientation of theorientation sensor 12000 in the camera coordinate system.

In step S10210, it is determined whether markers have been detected inthe input image. In the case that markers have been detected, the flowproceeds to step S10220. Otherwise, the flow proceeds to step S10240.

In step S10220, the sensor output value obtained by the sensor-outputobtaining unit 10040 is input to the image determination unit 10050.

In step S10230, an evaluation value for determining whether to use theinput image to calculate the orientation of the orientation sensor 12000in the camera coordinate system is calculated.

In general, angle information obtained only by a gyro sensor representsa relative orientation change with respect to an orientation at acertain time. An orientation sensor including the gyro sensor measuresthe direction of the earth's gravitational force using an accelerationsensor, thereby obtaining absolute tilt angles (pitch angle and rollangle) with reference to the direction of the gravitational force. Incontrast, an absolute reference cannot be obtained regarding an azimuthserving as a rotation around the gravity axis, and hence a drift errorcannot be corrected. The measurement accuracy of the azimuth is lowerthan that of the tilt angles. Therefore, using the markers in the image,an azimuth (yaw angle) drift error correction value and the orientationof the orientation sensor 12000 relative to the image pickup unit 11000are estimated. Whether to use the captured image for calculation isdetermined on whether or not the output value of the orientation sensor12000 changes by an amount greater than a certain threshold, that is,whether or not the image has been captured with a sufficiently differentorientation.

A method of calculating an evaluation value for determining whether theorientation is sufficiently different will be described below.

Images already determined to be used are numbered frame numbers 1, 2, .. . , m−1, and the obtained image serves as an m-th frame.

Let us calculate the angular difference between a sensor measurementvalue R_(m) at the time the image has been captured and a sensormeasurement value R_(j) at the time the j-th frame (j=1, 2, . . . , m−1)already determined to be used has been captured. First, the differencebetween these two orientations is calculated, i.e., ΔR=R_(m)·R_(j) ⁻¹.Then, ΔR, which is a 3×3 rotation transform matrix, is transformed toEuler angles. The angular difference is expressed in terms of the Eulerangles θ=[α β γ] around the x-axis, y-axis, and z-axis:

$\begin{matrix}\begin{matrix}{R = {R_{roll} \cdot R_{pitch} \cdot R_{yaw}}} \\{= {{\begin{bmatrix}{\cos\;\gamma} & {{- \sin}\;\gamma} & 0 \\{\sin\;\gamma} & {\cos\;\gamma} & 0 \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\alpha} & {{- \sin}\;\alpha} \\0 & {\sin\;\alpha} & {\cos\;\alpha}\end{bmatrix}}\begin{bmatrix}{\cos\;\beta} & 0 & {{- \sin}\;\beta} \\0 & 1 & 0 \\{\sin\;\beta} & 0 & {\cos\;\beta}\end{bmatrix}}}\end{matrix} & \left( {7\text{-}1} \right)\end{matrix}$

Taking into consideration an azimuth drift in the orientation sensor12000, the azimuth β regarded to contain a drift component is excluded,and only tilt angle components are extracted. More specifically, Eulerangle representation θ′=[α 0 γ] containing only tilt angle components,in which the azimuth component of θ is set to zero, is obtained, and arotation angle Δφ of the case in which the obtained θ′ is transformed torotation-axis rotation-angle representation is obtained. Thus, Δφ servesas an evaluation value.

In step S10240, whether to use the obtained image is determined on thebasis of the obtained evaluation value. Regarding all images (j=1, 2, .. . , J) that have already been determined to be used, in the case that:Δφ_(mj)>φ_(threshold)  (7-2)then, the obtained image is an image that has been captured with anorientation with which no image has been captured yet, and hence it isdetermined to use that image. In expression (7-2), φ_(threshold) is apreset threshold.

In step S10250, the determination result obtained in step S10240 isoutput to the orientation-sensor-position/orientation calculator 10060.

Referring back to FIG. 19, in step S10150, in the case that the image isdetermined in step S10140 to be used to calculate the orientation of theorientation sensor 12000, and that the number m of images that have beendetermined to be used is greater than or equal to two (m≧2), the flowproceeds to step S10160.

In contrast, in the case that the image is determined in step S10140 notto be used, the flow returns to step S10110.

In step S10160, using the images determined by the image determinationunit 10050 to be used, which have been captured at a plurality ofviewpoint positions, the orientation-sensor-position/orientationcalculator 10060 calculates the orientation ω_(CS) of the orientationsensor 12000 in the camera coordinate system. The position of the imagepickup unit 11000, an azimuth drift error correction value φτ, andω_(CS) are obtained by non-linear optimization so as to minimize theerror between the calculated position of each marker in the image, whichis calculated on the basis of the position of the orientation sensor12000, the sensor measurement value, the azimuth drift error correctionvalue φτ, and ω_(CS), and the detected position of each marker in theimage. Since this deriving method is similar to step S8060 described inthe second embodiment, a description thereof is omitted to avoidredundancy.

Finally in step S10170, it is determined whether to end the calculation.When the operator gives an instruction to end the calculation, the flowends. When the operator gives an instruction to continue the calculation(recalibration), an image is obtained again. In the case that a virtualobject rendered in the image using the obtained ω_(sc) and φτ iscorrectly registered with a corresponding real object, the operator endsthe calculation. Otherwise, the operator determines to continue thecalculation.

In the above manner, images appropriate to calculate the orientation ofthe orientation sensor 12000 mounted on the image pickup unit 11000 areautomatically obtained, and placement information regarding theorientation sensor 12000 can be highly accurately measured withoutdepending on the user's knowledge and skills.

Modification 7-1

In the seventh embodiment, the azimuth component is excluded whencalculating the evaluation value for determining whether to use theimage, and only the tilt angle components are used to conduct thedetermination. However, if the orientation sensor 12000 can performmeasurement with relatively high accuracy such that the effect of thedrift error can be ignored, the evaluation value may be calculatedwithout excluding the azimuth component, and whether to use the image tocalculate placement information regarding the orientation sensor 12000mounted on the image pickup unit 11000 may be determined.

An unknown parameter in the case that the azimuth drift is ignored whencalculating the placement information regarding the orientation sensor12000 mounted on the image pickup unit 11000 will be described below.

Given a vector l=(l₁, l₂, l₃) representing a vertical ascendingdirection (opposite to the earth's gravity) in the world coordinatesystem and an azimuth correction angle φ_(WT) representing an azimuthdifference angle (rotation angle around the axis in the direction ofgravitational force (gravity axis)) between the sensor coordinate systemand the world coordinate system. In this case, the sensor coordinatesystem is a coordinate system defined by the sensor, and the sensoroutput value represents the orientation in the sensor coordinate system.Thus, a coordinate transform must be applied to the orientation of ameasurement target in the sensor coordinate system to transform theorientation from the sensor coordinate system to the world coordinatesystem. This coordinate transform using the azimuth correction angleφ_(WT) may be performed using the method described in Japanese PatentLaid-Open No. 2005-107248, which has been applied by the assignee of thepresent invention.

The orientation ω_(CS) of the orientation sensor 12000 in the cameracoordinate system is processed as a three-element vector ω_(CS)=[ξψξ]^(T). The orientation ω_(CS) of the orientation sensor 12000 in thecamera coordinate system, the azimuth correction value φ_(WT), and theposition of the image pickup unit 11000 at a certain viewpoint position(represented by the identifier τ) t_(WCτ)=[x_(tτ) y_(tτ) z_(tτ)]^(T) areunknown. These unknown parameters are expressed in a 4+3L-dimensionalstate vector form:s=[ω_(SC) ^(T)ω_(WT)t_(WC1) ^(T) . . . t_(WCτ) ^(T) . . . t_(WCL)^(T)]^(T)where L is the total of images captured at different viewpoints.

The unknown parameter s can be obtained in a manner similar to thatdescribed in the second embodiment.

Other Embodiments

The present invention may also be implemented by supplying a system oran apparatus with a storage medium (or a recording medium) havingrecorded thereon program code of software implementing the features ofthe above-described embodiments and allowing a computer (or a CPU or amicro-processing unit (MPU)) of the system or apparatus to read andexecute the program code stored in the storage medium. In this case, theprogram code itself, which is read from the storage medium, implementsthe features of the above-described embodiments, and the storage mediumhaving the program code stored thereon constitutes an embodiment of thepresent invention. The features of the above-described embodiments maybe implemented not only by allowing the computer to read and execute theprogram code, but also by allowing an operating system (OS) running onthe computer to execute, on the basis of instructions of the programcode, part or entirety of the actual processing, thereby implementingthe features of the above-described embodiments.

The program code read from the storage medium may be written into amemory included in a function expansion card installed in the computeror a function expansion unit connected to the computer, and a CPU or thelike of the function expansion card or unit may perform part or entiretyof the actual processing on the basis of instructions of the programcode, thereby implementing the features of the foregoing embodiments.

In the case that the present invention is applied to the storage mediumdescribed above, the storage medium stores program code corresponding tothe flowcharts described above.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims the benefit of Japanese Application No.2006-173627 filed Jun. 23, 2006, which is hereby incorporated byreference herein in its entirety.

1. An information processing method of determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the method comprising the steps of: obtaining a first image captured by the image pickup unit from a first viewpoint; extracting markers from the first image; obtaining a position of the first viewpoint; calibrating the information regarding the calibration target using the markers extracted from the first image; determining a representative position of the markers extracted from the first image based on placement information of the markers; obtaining a second image captured by the image pickup unit from a second viewpoint after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint; obtaining a position of the second viewpoint; determining, on the basis of a positional relationship between the position of the first viewpoint, the representative position of the markers and the position of the second viewpoint, whether to calibrate the information regarding the calibration target using the second image; and calibrating the information regarding the calibration target using the second image in the case that it is determined to calibrate the information regarding the calibration target using the second image.
 2. The information processing method according to claim 1, wherein the information regarding the calibration target includes at least one of the placement information regarding the markers, the orientation of the image pickup unit, a parameter of the image pickup unit, an orientation of an orientation sensor which is mounted on the image pickup unit in a camera coordinate system of the image pickup unit, and a position and orientation of a position and orientation sensor which is mounted on the image pickup unit in the camera coordinate system of the image pickup unit.
 3. The information processing method according to claim 1, wherein the markers are identifiable markers.
 4. The information processing method according to claim 1, wherein the representative position is a barycentric position of the markers extracted from the first image.
 5. The information processing method according to claim 1, further comprising the steps of: calculating, on the basis of the position information of the image pickup unit at the first viewpoint, area information regarding an image capturing area in which the second image should be captured; and presenting the calculated area information.
 6. The information processing method according to claim 5, wherein an area subtending an angle greater than or equal to a predetermined angle relative to a straight line extending from a position represented by the position information of the image pickup unit at the first viewpoint to a representative position of the markers extracted from the captured image is calculated.
 7. The information processing method according to claim 1, wherein the determining step includes: calculating an evaluation value including a value of an angular difference between a vector given from the position of the first viewpoint to the representative position and a vector given from the position of the second viewpoint to the representative position; and determining to use the second image for calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.
 8. The information processing method according to claim 1, wherein the determining whether to calibrate further comprises: calculating an evaluation value including a value of a magnitude ratio between a vector given from the position of the first viewpoint to the representative position and a vector given from the position of the second viewpoint to the representative position; and determining to use the second image for calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.
 9. The information processing method according to claim 1, wherein, in the step of determining the representative position based on placement information of the markers extracted from the first image, the markers are weighted in accordance with a result of the extracted markers.
 10. The information processing method according to claim 1, wherein the representative position is on a visual axis of the image pickup unit.
 11. The information processing method according to claim 1, wherein obtaining the position information of the image pickup unit at the first and second viewpoints by using a position and orientation sensor.
 12. The information processing method according to claim 1, wherein obtaining the position information of the image pickup unit at the first and second viewpoints by using the placement information of the markers.
 13. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using an orientation sensor mounted onto the image pickup unit, wherein the information regarding the calibration target includes an orientation of the orientation sensor in a camera coordinate system of the image pickup unit.
 14. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using an orientation sensor mounted on the image pickup unit, wherein the information regarding the calibration target includes an orientation of the orientation sensor in a camera coordinate system of the image pickup unit.
 15. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using a position and orientation sensor mounted on the image pickup unit, wherein the information regarding the calibration target includes an orientation of the orientation sensor in a reference coordinate system.
 16. The information processing method according to claim 1, wherein the representative position is on a visual axis of the image pickup unit; the method further comprising: calculating an evaluation value including a value of an angular difference between a vector given from a visual axis of the image pickup unit at the first viewpoint and a vector given from a visual axis of the image pickup unit at the second viewpoint; and wherein, in the calibration determining step, the second image is determined for use in calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.
 17. The information processing method according to claim 1, further comprising: obtaining orientation information of the image pickup unit at the first viewpoint, calibrating the information regarding the calibration target using the markers extracted from the first image and the obtained position and orientation information of the image pickup unit at the first viewpoint; obtaining orientation information of the image pickup unit at the second viewpoint; and calibrating the information regarding the calibration target using the second image and the obtained position and orientation information of the image pickup unit at the first viewpoint, in the case that it is determined to calibrate the information regarding the calibration target using the second image.
 18. An information processing apparatus for determining whether to calibrate information of a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the apparatus comprising: a captured image obtaining unit configured to obtain a first image captured by the image pickup unit from a first viewpoint, and to obtain a second image captured by the image pickup unit from a second viewpoint; an extraction unit configured to extract markers from the first image, and to extract markers from the second image; an obtaining unit configured to obtain a position of the first viewpoint, and to obtain a position of the second viewpoint after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint; a representative position determination unit configured to calculate a representative position of the markers extracted from the first image based on placement information of the markers; a calibration determination unit configured to determine, on the basis of a positional relationship between the position of the first viewpoint, the representative position of the markers, and the position of the second viewpoint, whether to calibrate the information regarding the calibration target using the second image; and a calibration unit configured to calibrate the information regarding the calibration target using the markers extracted from the first image, and to calibrate the information regarding the calibration target using the second image in the case that the calibration determination unit determines to calibrate the information regarding the calibration target using the second image.
 19. An information processing method of determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the method comprising the steps of: obtaining a first image captured by the image pickup unit from a first viewpoint with a first visual axis; extracting markers from the first image; obtaining an orientation of the first visual axis; calibrating the information regarding the calibration target using the markers extracted from the first image; obtaining a second image captured by the image pickup unit from the second viewpoint with a second visual axis after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint; obtaining an orientation of the second visual axis; determining, based on an orientational relationship between the orientation of the first visual axis and the orientation of the second visual axis, whether to calibrate the information regarding the calibration target using the second image; and calibrating the information regarding the calibration target using the second image in the case that it is determined to calibrate the information regarding the calibration target using the second image.
 20. The information processing method according to claim 19, further comprising: calculating an angular difference between a vector given from a visual axis of the image pickup unit at the first viewpoint and a vector given from a visual axis of the image pickup unit at the second viewpoint, on the basis of the position and orientation of the image pickup nit at first viewpoint and the position and orientation of the image pickup unit at the second viewpoint; wherein, in the step of determining whether to calibrate, the second image is determined to use for calibrating the information regarding the calibration target, in the case that the angular difference is more than a threshold value set in advance.
 21. An information processing apparatus for determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the apparatus comprising: a captured image obtaining unit configured to obtain a first image captured by the image pickup unit from a first viewpoint with a first visual axis, and to obtain a second image captured by the image pickup unit from a second viewpoint with a second visual axis; an extraction unit configured to extract markers from the first image, and to extract markers from the second image; an obtaining unit configured to obtain an orientation of the first visual axis, and to obtain an orientation of the second visual axis after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint; a determination unit configured to determine, based on an orientational relationship between the orientation of the first visual axis and the orientation of the second visual axis, whether to calibrate the information regarding the calibration target using the second image; and a calibration unit configured to calibrate the information regarding the calibration target using the markers extracted from the first image, and to calibrate the information regarding the calibration target using the second image in the case that the determination unit determines to calibrate the information regarding the calibration target using the second image.
 22. A non-transitory computer-readable storage medium storing a program which causes a computer to execute an information processing method according to claim
 1. 23. A non-transitory computer-readable storage medium storing a program which causes a computer to execute an information processing method according to claim
 19. 